List:Commits« Previous MessageNext Message »
From:Christopher Powers Date:March 3 2009 7:10am
Subject:bzr commit into mysql-6.0-falcon-team branch (christopher.powers:3046)
Bug#32838 Bug#33177 Bug#42651
View as plain text  
#At file:///home/cpowers/work/dev/dev-06/mysql/

 3046 Christopher Powers	2009-03-03
      Bug #42651 "Regression: falcon_bug_22169-big started to fail with error 305"
      Bug #33177 "Table creation fails after error 305 and tablespace change"
      Bug #32838 "Falcon; error 1296 : Got error 305 'record memory is exhausted'"
      
      The fix for these bugs is the first of several improvements
      to Falcon's memory management (Worklog TBD).
      
      Falcon out-of-memory errors are caused by a combination of things.
      Recent improvements to the Scavenger and to the Backlogging subsystem
      (Bug#42592) have contributed to the resolution of these bugs, however,
      certain operations can still fill the record cache to the point where
      scavenging is ineffective.
      
      Scavenging efficiency will be greatly improved by allocating record
      data and metadata separately. The record cache now stores only
      actual record data, and Record and RecordVersion objects (metadata)
      are allocated from separate memory pools.
      
      The metadata memory pools are completely homogeneous, with no memory
      fragmentation. The record cache will also be far less fragmented,
      because large blocks of record data will no longer be interspersed
      with very small blocks of object data.
      
      Decoupling the data and metadata will also greatly reduce the number of
      out-of-memory conditions--typically seen during large inserts and
      updates--because the memory pools are allowed to grow independently.
      
      These memory pools may flucuate considerably during massive transactions,
      depending upon the record makeup and type of operation. This flucuation,
      however, serves only to emphasize the value managing these memory pools
      separately.
      
      One side-effect of this change is that, while the record cache max size
      remains fixed, the record metadata caches can grow unbounded. Although
      this is not unprecedented (Falcon's general purpose memory pool has
      always been unbounded), one remaining challenge is to ensure that
      the Falcon memory manager releases resources back to the system as
      soon as possible.
modified:
  storage/falcon/Database.cpp
  storage/falcon/Database.h
  storage/falcon/MemControl.cpp
  storage/falcon/MemControl.h
  storage/falcon/MemMgr.cpp
  storage/falcon/MemMgr.h
  storage/falcon/MemoryManager.h
  storage/falcon/RecordScavenge.cpp
  storage/falcon/RecordVersion.cpp
  storage/falcon/Scavenger.cpp
  storage/falcon/Table.cpp

per-file messages:
  storage/falcon/Database.cpp
    Access the memory managers via the MemControl class
    
    Database::Database()
    - Set pointers to the memory managers for record data and
    record metadata (Record and RecordVersion objects)
    
    Database::scavengeRecords()
    - Check scavenge threshold against active memory in record data pool
    
    Database::setLowMemory()
    - Added spaceNeeded parameter
  storage/falcon/Database.h
    Increased SCAVENGE_WAIT_MS from 20 to 50 ms
    
    Added MemMgr pointers: recordDataPool, recordPool and recordVersionPool
    
    Added Database::lowMemoryCount - max scavenge cycles for low memory to stay
    true below threshold
  storage/falcon/MemControl.cpp
    MemControl
    - ::maxMemory overrides combined memory limits of pools controlled
    by MemControl object
    - Added getCurrentMemory(). Can get current memory of combined
    pools within MemControl, or individual pools
  storage/falcon/MemControl.h
    Added getCurrentMemory()
  storage/falcon/MemMgr.cpp
    Defined memory managers for record data, record objects and record
    version objects.
    
    Defined unique memory managers ids for the separate memory pools.
  storage/falcon/MemMgr.h
    MemMgr
    - Added id
    - Added mgrId parameter to constructor
    - Added maxMemory to allow a memory limit for individual pools
  storage/falcon/MemoryManager.h
    Added MemMgr identifiers
    
    Defined two MemControl objects: one for general purpose
    memory control, the other for record data and metadata
  storage/falcon/RecordScavenge.cpp
    Simplified RecordScavenge::computeThreshold()
    
    Set startingActiveMemory to that of the record data pool
  storage/falcon/RecordVersion.cpp
    Comments
  storage/falcon/Scavenger.cpp
    Scavenger::execute()
    - Set 'forced' flag during low memory
  storage/falcon/Table.cpp
    Allocate record objects from the record object pool
    
    Allocate recordVersion objects from the record version object pool
=== modified file 'storage/falcon/Database.cpp'
--- a/storage/falcon/Database.cpp	2009-02-26 20:04:31 +0000
+++ b/storage/falcon/Database.cpp	2009-03-03 07:09:29 +0000
@@ -1,4 +1,4 @@
-/* Copyright � 2006-2008 MySQL AB, 2008-2009 Sun Microsystems, Inc.
+/* Copyright (C) 2006-2008 MySQL AB, 2008-2009 Sun Microsystems, Inc.
 
    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
@@ -72,6 +72,7 @@
 #include "InfoTable.h"
 #include "MemoryManager.h"
 #include "MemMgr.h"
+#include "MemControl.h"
 #include "RecordScavenge.h"
 #include "RecordSection.h"
 #include "LogStream.h"
@@ -117,6 +118,7 @@ static const char THIS_FILE[]=__FILE__;
 
 #define STATEMENT_RETIREMENT_AGE	60
 #define RECORD_RETIREMENT_AGE		60
+#define MAX_LOW_MEMORY				10
 
 extern uint falcon_debug_trace;
 
@@ -484,8 +486,10 @@ Database::Database(const char *dbName, C
 	scavengeCycle = 0;
 	serialLogBlockSize = configuration->serialLogBlockSize;
 	longSync = false;
-	recordDataPool = MemMgrGetFixedPool(MemMgrPoolRecordData);
-	//recordObjectPool = MemMgrGetFixedPool(MemMgrPoolRecordObject);
+	recordMemoryControl = MemMgrGetControl(MemMgrControlRecord);
+	recordPool = MemMgrGetFixedPool(MemMgrRecord);
+	recordVersionPool = MemMgrGetFixedPool(MemMgrRecordVersion);
+	recordDataPool = MemMgrGetFixedPool(MemMgrRecordData);
 	syncObject.setName("Database::syncObject");
 	syncTables.setName("Database::syncTables");
 	syncStatements.setName("Database::syncStatements");
@@ -497,7 +501,6 @@ Database::Database(const char *dbName, C
 	IO::deleteFile(BACKLOG_FILE);
 }
 
-
 void Database::start()
 {
 	symbolManager = new SymbolManager;
@@ -1826,7 +1829,7 @@ void Database::scavengeRecords(bool forc
 	// Take inventory of the record cache and prune invisible record versions
 
 	pruneRecords(&recordScavenge);
-	recordScavenge.prunedActiveMemory = recordDataPool->activeMemory;
+	recordScavenge.prunedActiveMemory = recordMemoryControl->getCurrentMemory(MemMgrRecordData);
 	recordScavenge.pruneStop = deltaTime;
 	syncScavenger.unlock();  // take a breath!
 
@@ -1834,15 +1837,20 @@ void Database::scavengeRecords(bool forc
 
 	syncScavenger.lock(Exclusive);
 	retireRecords(&recordScavenge);
-	recordScavenge.retiredActiveMemory = recordDataPool->activeMemory;
+	recordScavenge.retiredActiveMemory = recordMemoryControl->getCurrentMemory(MemMgrRecordData);
 	recordScavenge.retireStop = deltaTime;
 
 	// Enable backlogging if memory is low
 
 	if (recordScavenge.retiredActiveMemory > recordScavengeFloor)
-		setLowMemory();
+		if (!lowMemory)
+			setLowMemory(recordScavenge.retiredActiveMemory - recordScavengeFloor);
 	else
-		clearLowMemory();
+		{
+		if (lowMemoryCount)
+			if (--lowMemoryCount == 0)
+				clearLowMemory();
+		}
 
 	recordScavenge.print();
 	// Log::log(analyze(analyzeRecordLeafs));
@@ -1859,7 +1867,7 @@ void Database::scavengeRecords(bool forc
 	Sync syncMem(&syncMemory, "Database::checkRecordScavenge");
 	syncMem.lock(Exclusive);
 
-	lastActiveMemoryChecked = lastGenerationMemory = recordDataPool->activeMemory;
+	lastActiveMemoryChecked = lastGenerationMemory = recordMemoryControl->getCurrentMemory(MemMgrRecordData);
 }
 
 // Take inventory of the record cache and prune invisible record versions
@@ -1892,7 +1900,7 @@ void Database::retireRecords(RecordScave
 	// Scavenge if we passed the upper limit or if a forced scavenge
 	// was requested.
 
-	if (recordDataPool->activeMemory < recordScavengeThreshold
+	if (recordMemoryControl->getCurrentMemory(MemMgrRecordData) < recordScavengeThreshold
 		&& !recordScavenge->forced)
 		return;
 
@@ -1902,7 +1910,7 @@ void Database::retireRecords(RecordScave
 	Sync syncTbl(&syncTables, "Database::retireRecords(2)");
 	syncTbl.lock(Shared);
 
-	uint64 spaceToRetire = recordDataPool->activeMemory - recordScavengeFloor;
+	uint64 spaceToRetire = recordMemoryControl->getCurrentMemory(MemMgrRecordData) - recordScavengeFloor;
 	recordScavenge->computeThreshold(spaceToRetire);
 
 	for (Table *table = tableList; table; table = table->next)
@@ -1956,7 +1964,7 @@ void Database::scavengerThreadMain(void)
 		{
 		scavenge((scavengeForced > 0));
 		
-		if (recordDataPool->activeMemory < recordScavengeThreshold)
+		if (recordMemoryControl->getCurrentMemory(MemMgrRecordData) < recordScavengeThreshold)
 			{
 			INTERLOCKED_INCREMENT(scavengerThreadSleeping);
 			thread->sleep();
@@ -2611,32 +2619,32 @@ void Database::checkRecordScavenge(void)
 		syncMem.lock(Exclusive);
 
 		if (   !scavengerThreadSignaled 
-			&& (recordDataPool->activeMemory > lastActiveMemoryChecked))
+			&& (recordMemoryControl->getCurrentMemory(MemMgrRecordData) > lastActiveMemoryChecked))
 			{
 			// Start a new age generation regularly.  Note that since activeMemory
 			// can go down due to a recent scavenge, it is possible for
-			// lastGenerationMemory to be > recordDataPool->activeMemory
+			// lastGenerationMemory to be > recordMemoryControl->getCurrentMemory()
 
-			if (  (int64) (recordDataPool->activeMemory - lastGenerationMemory)
+			if (  (int64) (recordMemoryControl->getCurrentMemory(MemMgrRecordData) - lastGenerationMemory)
 			    > (int64) recordScavengeMaxGroupSize)
 				{
 				// Let the scavenger run to prune records.  
 				// It will also retire records if recordScavengeThreshold has been reached.
 
 				INTERLOCKED_INCREMENT (currentGeneration);
-				lastGenerationMemory = recordDataPool->activeMemory;
+				lastGenerationMemory = recordMemoryControl->getCurrentMemory(MemMgrRecordData);
 
 				INTERLOCKED_INCREMENT(scavengerThreadSignaled);
 				scavengerThreadWakeup();
 				}
 
-			else if (recordDataPool->activeMemory >= recordScavengeThreshold)
+			else if (recordMemoryControl->getCurrentMemory(MemMgrRecordData) >= recordScavengeThreshold)
 				{
 				INTERLOCKED_INCREMENT(scavengerThreadSignaled);
 				scavengerThreadWakeup();
 				}
 
-			lastActiveMemoryChecked = recordDataPool->activeMemory;
+			lastActiveMemoryChecked = recordMemoryControl->getCurrentMemory(MemMgrRecordData);
 			}
 		}
 }
@@ -2757,7 +2765,7 @@ void Database::flushWait(void)
 	cache->flushWait();
 	}
 
-void Database::setLowMemory(void)
+void Database::setLowMemory(uint64 spaceNeeded)
 {
 	if (!backLog)
 		{	
@@ -2769,9 +2777,12 @@ void Database::setLowMemory(void)
 		}
 
 	lowMemory = true;
+	lowMemoryCount = MAX_LOW_MEMORY;
+//	this->transactionManager->setLowMemory(spaceNeeded);
 }
 
 void Database::clearLowMemory(void)
 {
+//	this->transactionManager->clearLowMemory();
 	lowMemory = false;
 }

=== modified file 'storage/falcon/Database.h'
--- a/storage/falcon/Database.h	2009-02-26 20:04:31 +0000
+++ b/storage/falcon/Database.h	2009-03-03 07:09:29 +0000
@@ -1,4 +1,4 @@
-/* Copyright � 2006-2008 MySQL AB, 2008-2009 Sun Microsystems, Inc.
+/* Copyright (C) 2006-2008 MySQL AB, 2008-2009 Sun Microsystems, Inc.
 
    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
@@ -61,7 +61,7 @@ static const int OUT_OF_RECORD_MEMORY_RE
 
 // Milliseconds per iteration to wait for the Scavenger
 
-static const int SCAVENGE_WAIT_MS			  = 20;
+static const int SCAVENGE_WAIT_MS			  = 50;//500;
 
 // Scavenger cycles per call to updateCardinalities()
 
@@ -120,6 +120,7 @@ class IndexKey;
 class InfoTable;
 class TableSpace;
 class MemMgr;
+class MemControl;
 class RecordScavenge;
 class PriorityScheduler;
 class SQLException;
@@ -255,7 +256,7 @@ public:
 	void			setIOError(SQLException* exception);
 	void			clearIOError(void);
 	void			flushWait(void);
-	void			setLowMemory(void);
+	void			setLowMemory(uint64 spaceNeeded);
 	void			clearLowMemory(void);
 
 
@@ -332,7 +333,10 @@ public:
 	volatile INTERLOCK_TYPE	scavengeForced;
 	PageWriter			*pageWriter;
 	PreparedStatement	*updateCardinality;
-	MemMgr				*recordDataPool;
+	MemControl			*recordMemoryControl;
+	MemMgr				*recordDataPool;	// Record data pool (no object metadata)
+	MemMgr				*recordPool;		// Record object pool
+	MemMgr				*recordVersionPool;	// RecordVersion object pool
 	time_t				startTime;
 	
 	volatile int		deltaTime;
@@ -356,6 +360,7 @@ public:
 	uint64				lastGenerationMemory;
 	uint64				lastActiveMemoryChecked;
 	uint64				scavengeCount;
+	uint64				lowMemoryCount;
 	time_t				creationTime;
 	volatile time_t		lastScavenge;
 };

=== modified file 'storage/falcon/MemControl.cpp'
--- a/storage/falcon/MemControl.cpp	2008-05-14 18:39:57 +0000
+++ b/storage/falcon/MemControl.cpp	2009-03-03 07:09:29 +0000
@@ -20,20 +20,53 @@
 MemControl::MemControl(void)
 {
 	count = 0;
+	maxMemory = 0;
 }
 
 MemControl::~MemControl(void)
 {
 }
 
+// Verify that the requested memory will not exceed the memory limit.
+//
+// If a limit has not been set for the group of memory pools, compare
+// against the total of individual pool limits.
+//
+// Note that allocation may occur during static initialization,
+// before the MemControl has been fully initialized and the memory
+// limits set.
+
 bool MemControl::poolExtensionCheck(uint size)
 {
 	uint64 inUse = size;
 	
-	for (int n = 0; n < count; ++n)
-		inUse += pools[n]->currentMemory;
+	// If non-zero, MemControl::maxMemory supercedes the
+	// total of the individual pool limits.
+	
+	if (maxMemory)
+		{
+		for (int n = 0; n < count; ++n)
+			inUse += pools[n]->currentMemory;
+			
+		return inUse < maxMemory;
+		}
+	else
+		{
+		// If no group maximum is set, total the individual pool
+		// limits. Only consider pools for which a limit has been set.
+		
+		uint64 poolMax = 0;
+		for (int n = 0; n < count; ++n)
+			{
+			if (pools[n]->maxMemory)
+				{
+				inUse += pools[n]->currentMemory;
+				poolMax += pools[n]->maxMemory;
+				}
+			}
 	
-	return inUse < maxMemory;
+		return poolMax ? (inUse < poolMax) : true;
+		}
 }
 
 void MemControl::addPool(MemMgr* pool)
@@ -42,7 +75,48 @@ void MemControl::addPool(MemMgr* pool)
 	pool->memControl = this;
 }
 
+// Set the memory limit for the group of pools
+
 void MemControl::setMaxSize(uint64 size)
 {
 	maxMemory = size;
 }
+
+// Set the memory limit for a specific pool. Note that the
+// size is only set for the first pool matching mgrId.
+
+void MemControl::setMaxSize(int mgrId, uint64 size)
+{
+	uint64 groupTotal = 0;
+	
+	for (int n = 0; n < count; ++n)
+		groupTotal += pools[n]->maxMemory;
+
+	// Set the memory limit after checking for overflow
+		
+	for (int n = 0; n < count; ++n)
+		{
+		if (pools[n]->id == mgrId)
+			{
+			uint64 remaining = MaxTotalMemory - groupTotal;
+			pools[n]->maxMemory = (size < remaining ? size : remaining);
+			break;
+			}
+		}
+}
+
+// Total memory in use for the pools in this group,
+// specified by poolMask. Default is all pools.
+
+uint64 MemControl::getCurrentMemory(int poolMask)
+{
+	uint64 inUse = 0;
+	
+	for (int n = 0; n < count; ++n)
+		if (pools[n]->id & poolMask)
+			inUse += pools[n]->currentMemory;
+	
+	return inUse;
+}
+
+

=== modified file 'storage/falcon/MemControl.h'
--- a/storage/falcon/MemControl.h	2008-05-14 18:39:57 +0000
+++ b/storage/falcon/MemControl.h	2009-03-03 07:09:29 +0000
@@ -29,8 +29,10 @@ public:
 	int		count;
 	uint64	maxMemory;
 	virtual bool poolExtensionCheck(uint size);
+	virtual uint64 getCurrentMemory(int poolMask = MemMgrAllPools);
 	void addPool(MemMgr* pool);
 	void setMaxSize(uint64 size);
+	void setMaxSize(int mgrId, uint64 size);
 };
 
 #endif

=== modified file 'storage/falcon/MemMgr.cpp'
--- a/storage/falcon/MemMgr.cpp	2009-02-23 22:42:36 +0000
+++ b/storage/falcon/MemMgr.cpp	2009-03-03 07:09:29 +0000
@@ -35,6 +35,7 @@
 #include "Stream.h"
 #include "InfoTable.h"
 #include "MemControl.h"
+#include "RecordVersion.h"
 
 #ifdef HAVE_purify
 #ifdef HAVE_CONFIG
@@ -79,10 +80,18 @@ const int validateMinutia	= 16;
 // Nominal memory limits at startup--final values set during initialization
 bool memoryManagerAlive;
 
-static MemMgr		memoryManager(defaultRounding, FREE_OBJECTS_SIZE, HEAP_SIZE,&memoryManagerAlive);
-static MemMgr		recordManager(defaultRounding, 2, HEAP_SIZE);
-//static MemMgr		recordObjectManager (defaultRounding, sizeof(RecordVersion) + 100, HEAP_SIZE);
-static MemControl	memControl;
+static MemControl memControlGeneral;
+static MemControl memControlRecord;
+
+// General purpose memory
+static MemMgr memoryManager(MemMgrGeneral, defaultRounding, FREE_OBJECTS_SIZE, HEAP_SIZE,&memoryManagerAlive, &memControlGeneral);
+
+// Record data
+static MemMgr recordDataManager(MemMgrRecordData, defaultRounding, 2, HEAP_SIZE, NULL, &memControlRecord);
+
+// Record metadata: Record object and RecordVersion object 
+static MemMgr recordManager(MemMgrRecord, defaultRounding, sizeof(RecordVersion) + 100, HEAP_SIZE, NULL, &memControlRecord);
+static MemMgr recordVersionManager(MemMgrRecordVersion, defaultRounding, sizeof(RecordVersion) + 100, HEAP_SIZE, NULL, &memControlRecord);
 
 #ifdef _DEBUG
 static void		*stopAddress;
@@ -161,12 +170,12 @@ struct Client {
 
 	void* MemMgrRecordAllocate (size_t size, const char *file, int line)
 	{
-		return recordManager.allocateDebug (size, file, line);
+		return recordDataManager.allocateDebug (size, file, line);
 	}
 
 	void MemMgrRecordDelete (char *record)
 	{
-		recordManager.releaseDebug (record);
+		recordDataManager.releaseDebug (record);
 	}
 #else
 	void* MemMgrPoolAllocate (MemMgr *pool, size_t s)
@@ -192,12 +201,12 @@ struct Client {
 
 	void* MemMgrRecordAllocate (size_t size, const char *file, int line)
 	{
-		return recordManager.allocate (size);
+		return recordDataManager.allocate (size);
 	}
 
 	void MemMgrRecordDelete (char *record)
 	{
-		recordManager.release (record);
+		recordDataManager.release (record);
 	}
 
 #endif
@@ -205,8 +214,9 @@ struct Client {
 void MemMgrValidate ()
 {
 	memoryManager.validate();
+	recordDataManager.validate();
 	recordManager.validate();
-	//recordObjectManager.validate();
+	recordVersionManager.validate();
 }
 
 void MemMgrValidate (void *object)
@@ -221,7 +231,7 @@ void MemMgrAnalyze(int mask, Stream *str
 	stream->putSegment ("Memory\n");
 	memoryManager.analyze (mask, stream, NULL, NULL);
 	stream->putSegment ("Records\n");
-	recordManager.analyze (mask, stream, NULL, NULL);
+	recordDataManager.analyze (mask, stream, NULL, NULL);
 	//LEAVE_CRITICAL_SECTION;
 }
 
@@ -238,47 +248,61 @@ void MemMgrAnalyze(MemMgrWhat what, Info
 			break;
 
 		case MemMgrRecordSummary:
+			recordDataManager.analyze(0, NULL, infoTable, NULL);
 			recordManager.analyze(0, NULL, infoTable, NULL);
-			//recordObjectManager.analyze(0, NULL, infoTable, NULL);
+			recordVersionManager.analyze(0, NULL, infoTable, NULL);
 			break;
 
 		case MemMgrRecordDetail:
+			recordDataManager.analyze(0, NULL, NULL, infoTable);
 			recordManager.analyze(0, NULL, NULL, infoTable);
-			//recordObjectManager.analyze(0, NULL, NULL, infoTable);
+			recordVersionManager.analyze(0, NULL, NULL, infoTable);
 			break;
 		}
 }
 
+// Set memory limit for the record data pool
+
 void MemMgrSetMaxRecordMember (long long size)
 {
-	if (!recordManager.memControl)
-		{
-		memControl.addPool(&recordManager);
-		//memControl.addPool(&recordObjectManager);
-		}
-	memControl.setMaxSize(size);
+	memControlRecord.setMaxSize(MemMgrRecordData, size);
 }
 
 MemMgr*	MemMgrGetFixedPool (int id)
 {
 	switch (id)
 		{
-		case MemMgrPoolGeneral:
+		case MemMgrGeneral:
 			return &memoryManager;
 
-		case MemMgrPoolRecordData:
-			return &recordManager;
+		case MemMgrRecordData:
+			return &recordDataManager;
 
-		/***
-		case MemMgrPoolRecordObject:
-			return &recordObjectManager;
-		***/
+		case MemMgrRecord:
+			return &recordManager;
 
+		case MemMgrRecordVersion:
+			return &recordVersionManager;
+			
 		default:
 			return NULL;
 		}
 }
 
+MemControl*	MemMgrGetControl (int id)
+{
+	switch (id)
+		{
+		case MemMgrControlGeneral:
+			return &memControlGeneral;
+
+		case MemMgrControlRecord:
+			return &memControlRecord;
+
+		default:
+			return NULL;
+		}
+}
 
 void MemMgrLogDump()
 {
@@ -288,23 +312,24 @@ void MemMgrLogDump()
 #endif
 }
 
-
-MemMgr::MemMgr(int rounding, int cutoff, int minAlloc, bool *alive) : mutex("MemMgr::mutex")
-{
-	signature = defaultSignature;
-	roundingSize = rounding;
-	threshold = cutoff;
-	minAllocation = minAlloc;
-	currentMemory		= 0;
-	blocksAllocated		= 0;
-	blocksActive		= 0;
-	activeMemory		= 0;
-	numberBigHunks		= 0;
-	numberSmallHunks	= 0;
-	bigHunks			= NULL;
-	smallHunks			= NULL;
-	memControl			= NULL;
-
+MemMgr::MemMgr(int mgrId, int rounding, int cutoff, int minAlloc, bool *alive, MemControl *memCtrl)
+				: id(mgrId), roundingSize(rounding), threshold(cutoff),  minAllocation(minAlloc), 
+				  memControl(memCtrl), mutex("MemMgr::mutex"), isAlive(alive)
+{
+	signature		 = defaultSignature;
+	activeMemory	 = 0;
+	currentMemory	 = 0;
+	maxMemory		 = 0;
+	blocksAllocated	 = 0;
+	blocksActive	 = 0;
+	numberBigHunks	 = 0;
+	numberSmallHunks = 0;
+	bigHunks		 = NULL;
+	smallHunks		 = NULL;
+	
+	if (memControl)
+		memControl->addPool(this);
+	
 	int vecSize = (cutoff + rounding) / rounding;
 	int l = vecSize * sizeof (void*);
 
@@ -313,14 +338,11 @@ MemMgr::MemMgr(int rounding, int cutoff,
 	//freeBlocks.nextLarger = freeBlocks.priorSmaller = &freeBlocks;
 	//freeBlockTree = NULL;
 	junk.larger = junk.smaller = &junk;
-	isAlive = alive;
+	
 	if(alive)
-	{
 		*alive = true;
-	}
 }
 
-
 MemMgr::MemMgr(void* arg1, void* arg2) : mutex("MemMgr::mutex")
 {
 	MemMgr();
@@ -576,7 +598,6 @@ void* MemMgr::allocateDebug(size_t size,
 	return &memory->body;
 }
 
-
 void MemMgr::release(void* object)
 {
 	if (object)

=== modified file 'storage/falcon/MemMgr.h'
--- a/storage/falcon/MemMgr.h	2009-01-08 09:05:26 +0000
+++ b/storage/falcon/MemMgr.h	2009-03-03 07:09:29 +0000
@@ -35,6 +35,10 @@ static const int defaultCutoff = 4096;
 static const int defaultAllocation = 65536;
 static const int defaultSignature = 12345678;
 
+// Limit of the total memory for the pools within a MemControl group
+
+static const uint64 MaxTotalMemory = 0xffffffffffffffffLL;
+
 class MemMgr;
 class Stream;
 class InfoTable;
@@ -107,11 +111,17 @@ public:
 class MemMgr
 {
 public:
-	MemMgr(int rounding=defaultRounding, int cutoff=defaultCutoff, 
-		int minAllocation=defaultAllocation, bool *alive = NULL);
+	MemMgr(	int mgrId = MemMgrDefault,
+			int rounding = defaultRounding,
+			int cutoff = defaultCutoff, 
+			int minAllocation = defaultAllocation,
+			bool *alive = NULL,
+			MemControl *memCtrl = NULL);
+	
 	MemMgr(void* arg1, void* arg2);
 	virtual ~MemMgr(void);
 
+	int				id;
 	int				signature;	
 	int				roundingSize;
 	int				threshold;
@@ -129,6 +139,7 @@ public:
 	Mutex			mutex;		// Win32 critical regions are faster than SyncObject
 	int64			currentMemory;
 	uint64			activeMemory;
+	uint64			maxMemory;
 	int				blocksAllocated;
 	int				blocksActive;
 	bool			*isAlive;

=== modified file 'storage/falcon/MemoryManager.h'
--- a/storage/falcon/MemoryManager.h	2008-10-31 15:42:42 +0000
+++ b/storage/falcon/MemoryManager.h	2009-03-03 07:09:29 +0000
@@ -40,6 +40,7 @@
 class Stream;
 class InfoTable;
 class MemMgr;
+class MemControl;
 struct MemObject;
 
 #ifdef _DEBUG
@@ -110,9 +111,19 @@ enum MemMgrWhat {
 	MemMgrRecordDetail
 	};
 
-static const int MemMgrPoolGeneral		= 0;
-static const int MemMgrPoolRecordData	= 1;
-static const int MemMgrPoolRecordObject	= 2;
+// Memory pool identifiers
+
+static const int MemMgrDefault		 = 1;
+static const int MemMgrGeneral		 = 2;
+static const int MemMgrRecordData	 = 4;
+static const int MemMgrRecord		 = 8;
+static const int MemMgrRecordVersion = 16;
+static const int MemMgrAllPools		 = -1;
+
+// Memory pool group identifiers
+
+static const int MemMgrControlGeneral = 0; // general memory pool
+static const int MemMgrControlRecord  = 1; // record and object memory pools
 
 extern void		MemMgrAnalyze(MemMgrWhat what, InfoTable *table);
 extern void		MemMgrRelease (void *object);
@@ -122,6 +133,7 @@ extern void*	MemMgrRecordAllocate (size_
 extern void		MemMgrRecordDelete (char *record);
 extern void		MemMgrSetMaxRecordMember (long long size);
 extern MemMgr*	MemMgrGetFixedPool (int id);
+extern MemControl* MemMgrGetControl (int id);
 
 extern MemObject* MemMgrFindPriorBlock (void *block);
 

=== modified file 'storage/falcon/RecordScavenge.cpp'
--- a/storage/falcon/RecordScavenge.cpp	2009-02-25 16:39:29 +0000
+++ b/storage/falcon/RecordScavenge.cpp	2009-03-03 07:09:29 +0000
@@ -21,6 +21,7 @@
 #include "RecordVersion.h"
 #include "Log.h"
 #include "MemMgr.h"
+#include "MemControl.h"
 #include "Sync.h"
 #include "Transaction.h"
 #include "TransactionManager.h"
@@ -34,7 +35,7 @@ RecordScavenge::RecordScavenge(Database 
 	veryOldRecords = 0;
 	veryOldRecordSpace = 0;
 
-	startingActiveMemory = database->recordDataPool->activeMemory;
+	startingActiveMemory = database->recordMemoryControl->getCurrentMemory(MemMgrRecordData);
 	prunedActiveMemory = 0;
 	retiredActiveMemory = 0;
 
@@ -239,7 +240,12 @@ Record* RecordScavenge::inventoryRecord(
 uint64 RecordScavenge::computeThreshold(uint64 spaceToRetire)
 {
 	uint64 totalSpace = veryOldRecordSpace;
-	scavengeGeneration = 0;
+	scavengeGeneration = baseGeneration;
+	
+	// Don't mess around if memory is critical
+	
+	if (database->lowMemory)
+		return scavengeGeneration;
 
 	// The baseGeneration is the currentGeneration when the scavenge started
 	// It is in ageGroups[0].  Next oldest in ageGroups[1], etc.
@@ -254,10 +260,7 @@ uint64 RecordScavenge::computeThreshold(
 			scavengeGeneration = baseGeneration - n;
 		}
 
-	// We still may want to scavenge even if the age group total is
-	// too small, so use the base generation as a starting point.
-	
-	return (scavengeGeneration ? scavengeGeneration : baseGeneration);
+	return scavengeGeneration;
 }
 
 void RecordScavenge::print(void)

=== modified file 'storage/falcon/RecordVersion.cpp'
--- a/storage/falcon/RecordVersion.cpp	2009-02-26 20:04:31 +0000
+++ b/storage/falcon/RecordVersion.cpp	2009-03-03 07:09:29 +0000
@@ -1,4 +1,4 @@
-/* Copyright � 2006-2008 MySQL AB, 2008-2009 Sun Microsystems, Inc.
+/* Copyright (C) 2006-2008 MySQL AB, 2008-2009 Sun Microsystems, Inc.
 
    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
@@ -68,6 +68,9 @@ RecordVersion::RecordVersion(Table *tbl,
 
 RecordVersion::RecordVersion(Database* database, Serialize *stream) : Record(database, stream)
 {
+	// Reconstitute a record version and recursively restore all
+	// prior versions from 'stream'
+
 	virtualOffset = stream->getInt64();
 	transactionId = stream->getInt();
 	int priorType = stream->getInt();
@@ -447,6 +450,8 @@ void RecordVersion::serialize(Serialize*
 	stream->putInt64(virtualOffset);
 	stream->putInt(transactionId);
 	
+	// Recursively serialize the prior version chain
+	
 	if (priorVersion)
 		{
 		stream->putInt(priorVersion->isVersion());

=== modified file 'storage/falcon/Scavenger.cpp'
--- a/storage/falcon/Scavenger.cpp	2009-01-08 09:05:26 +0000
+++ b/storage/falcon/Scavenger.cpp	2009-03-03 07:09:29 +0000
@@ -65,7 +65,8 @@ void Scavenger::scavenge()
 
 void Scavenger::execute(Scheduler * scheduler)
 {
-	database->signalScavenger();
+	bool forced = database->scavengeForced || database->lowMemory;
+	database->signalScavenger(forced);
 	getNextEvent();
 	scheduler->addEvent (this);
 }

=== modified file 'storage/falcon/Table.cpp'
--- a/storage/falcon/Table.cpp	2009-03-03 06:30:21 +0000
+++ b/storage/falcon/Table.cpp	2009-03-03 07:09:29 +0000
@@ -3669,7 +3669,7 @@ RecordVersion* Table::allocRecordVersion
 			if ((++database->recordPoolAllocCount & 0x7F) == 0)
 				database->checkRecordScavenge();
 
-			return POOL_NEW(database->recordDataPool) RecordVersion(this, format, transaction, priorVersion);
+			return POOL_NEW(database->recordVersionPool) RecordVersion(this, format, transaction, priorVersion);
 			}
 
 		catch (SQLException& exception)
@@ -3706,7 +3706,7 @@ Record* Table::allocRecord(int recordNum
 			if ((++database->recordPoolAllocCount & 0x7F) == 0)
 				database->checkRecordScavenge();
 
-			return POOL_NEW(database->recordDataPool) Record (this, recordNumber, stream);
+			return POOL_NEW(database->recordPool) Record (this, recordNumber, stream);
 			}
 
 		catch (SQLException& exception)

Thread
bzr commit into mysql-6.0-falcon-team branch (christopher.powers:3046)Bug#32838 Bug#33177 Bug#42651Christopher Powers3 Mar