From: Kevin Lewis Date: January 19 2009 4:58pm Subject: Re: When you have a chance... List-Archive: http://lists.mysql.com/falcon/392 Message-Id: <4974B122.3090800@sun.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="Boundary_(ID_7IU+CYIY9/gJL3OJz8Uweg)" --Boundary_(ID_7IU+CYIY9/gJL3OJz8Uweg) Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT Chris, I thought a lot about this solution the last couple of days. Using a timeout as a standard expected event in handling a concurrency issue is not the best approach, I'm sure Jim would agree. But there are extenuating circumstances. You have shown that we need to protect other threads that call hasData() from calling that concurrently with an active chill thread because it could be there in one instant and not the next. The act of chilling a record is very intrusive and dangerous without a record mutex. But using a record mutex is against Jim's MVCC philosophy for many reasons which need not be listed here. We just can't do it. So testing and using the syncPrior as an alternative to the record mutex is OK, I think. Especially since there is no hard requirement to chill any particular record. Since the action on the record is not required, and chilling should not happen very often, I think a timeout and move-on approach is acceptable here. Especially since I cannot think of a better approach! :) Jim, this is your invitation to spark a better idea, or think of a better approach. Note that this use of syncPrior during a chill actually works, even though it is not pretty. Kevin Christopher Powers wrote: > ...call me today 612-729-1519 or my cell, 612-616-1069. I'd like to > discuss the changes I did for garbage collect. > > I pushed the fixes to mysql-6.0-falcon-chris on PB2, and *both* > falcon_online_alter and falcon_chill_thaw passed. I find this to be very > encouraging. > > Using syncPrior in chill was not straightforward solution because of a > deadlock with a low-level thaw, but what I have works. > --Boundary_(ID_7IU+CYIY9/gJL3OJz8Uweg) Content-type: message/rfc822; name="Attached Message" Return-path: Received: from fe-amer-09.sun.com ([unknown] [192.18.109.79]) by amer3-mail1.central.sun.com (Sun Java(tm) System Messaging Server 7.0-3.01 64bit (built Dec 9 2008)) with ESMTP id <0KDL00KCIWQUUK70@stripped> for kl221630@stripped; Sat, 17 Jan 2009 01:25:42 -0700 (MST) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0KDL00301WQUY600@stripped> for kl221630@stripped (ORCPT klewis@stripped); Sat, 17 Jan 2009 01:25:42 -0700 (MST) Received: from phys-amer3-2.central.sun.com ([129.147.157.45]) by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTP id <0KDL007H1WQTTZ80@stripped> for kl221630@stripped (ORCPT klewis@stripped); Sat, 17 Jan 2009 01:25:41 -0700 (MST) Received: from dm-central-01.central.sun.com ([unknown] [129.147.62.4]) by amer3-mail1.central.sun.com (Sun Java(tm) System Messaging Server 7.0-3.01 64bit (built Dec 9 2008)) with ESMTP id <0KDL00KCGWQTUK70@stripped> for kl221630@stripped (ORCPT klewis@stripped); Sat, 17 Jan 2009 01:25:41 -0700 (MST) Received: from newsunmail1brm.central.sun.com (newsunmail1brm.Central.Sun.COM [129.147.62.245]) by dm-central-01.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n0H8PfkA009023 for ; Sat, 17 Jan 2009 01:25:41 -0700 (MST) Received: from nwk-avmta-1.SFBay.Sun.COM (nwk-avmta-1.SFBay.Sun.COM [129.146.11.74]) by newsunmail1brm.central.sun.com (8.13.7+Sun/8.13.7/ENSMAIL,v2.2) with ESMTP id n0H8PexZ037054 for <@sunmail2sca.sfbay.sun.com:kevin.lewis@stripped>; Sat, 17 Jan 2009 01:25:41 -0700 (MST) Received: from pmxchannel-daemon.nwk-avmta-1.sfbay.Sun.COM by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KDL00G0HWQSR200@stripped> for kevin.lewis@stripped (ORCPT klewis@stripped); Sat, 17 Jan 2009 00:25:40 -0800 (PST) Received: from dm-xlate-02.central.sun.com ([129.147.62.12]) by nwk-avmta-1.sfbay.Sun.COM (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KDL0051RWQRPY90@stripped> for kevin.lewis@stripped (ORCPT klewis@stripped); Sat, 17 Jan 2009 00:25:40 -0800 (PST) Received: from brmea-mail-1.sun.com (brmea-mail-1.Sun.COM [192.18.98.31]) by dm-xlate-02.central.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n0H8Pdhv048160 for ; Sat, 17 Jan 2009 01:25:39 -0700 (MST) Received: from relay11i.sun.com (ip121.net129179-4.block1.us.syntegra.com [129.179.4.121]) by brmea-mail-1.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n0H8Pdk7011960 for ; Sat, 17 Jan 2009 08:25:39 +0000 (GMT) Received: from mmp11es.mmp.us.syntegra.com ([160.41.208.11] [160.41.208.11]) by relay11i.sun.com with ESMTP id BT-MMP-2781583 for klewis@stripped; Sat, 17 Jan 2009 08:25:39 +0000 (Z) Received: from relay15i.sun.com (relay15i.sun.com [129.179.4.125]) by mmp11es.mmp.us.syntegra.com with ESMTP id BT-MMP-66436822 for klewis@stripped; Sat, 17 Jan 2009 08:25:38 +0000 (Z) Received: from lists.mysql.com ([213.136.52.31] [213.136.52.31]) by relay1i.sun.com id BT-MMP-26688437 for klewis@stripped; Sat, 17 Jan 2009 08:25:38 +0000 (Z) Received: (qmail 10394 invoked by uid 510); Sat, 17 Jan 2009 08:25:36 +0000 Received: (qmail 10369 invoked by uid 509); Sat, 17 Jan 2009 08:25:36 +0000 Date: Sat, 17 Jan 2009 02:23:04 -0600 (CST) From: Christopher Powers Subject: bzr commit into mysql-6.0-falcon-team branch (christopher.powers:2964) Bug#42131 Sender: Christopher.Powers@stripped To: commits@stripped Message-id: <20090117082305.6075E1DB074C@stripped> X-Envelope-from: Kevin.Lewis@stripped X-Envelope-to: falcon@stripped, jstarkey@stripped MIME-version: 1.0 Content-type: text/plain; charset=us-ascii Content-transfer-encoding: 7BIT Precedence: bulk Delivered-to: mailing list commits@stripped Received-SPF: pass (lists.mysql.com: local policy) X-PMX-Version: 5.4.1.325704 X-Brightmail-Tracker: AAAAAA== X-Spam-Check-By: lists.mysql.com X-Bug: 42131 X-CSetKey: christopher.powers@stripped X-bzr-action: commit X-bzr-revid: christopher.powers@stripped X-bzr-revno: 2964 X-Cxn-Txn: 66251338,46232658 X-Antispam: No, score=-0.7/5.0, scanned in 0.113sec at (localhost [127.0.0.1]) by smf-spamd v1.3.1 - http://smfs.sf.net/ Mailing-List: contact commits-help@stripped; run by ezmlm List-Post: List-Unsubscribe: List-Archive: http://lists.mysql.com/commits/63498 List-Help: List-Id: User-Agent: Bazaar (1.5) Original-recipient: rfc822;klewis@stripped X-Spam-Status: No, hits=0.0 required=5.0 tests=UNPARSEABLE_RELAY #At file:///home/cpowers/work/dev/dev-03/mysql/ 2964 Christopher Powers 2009-01-17 Bug #42131 "falcon_backlog test crashes in Record::getEncodedValue" Transaction:chill() must get an exclusive syncPrior lock before deleting record data. modified: storage/falcon/Record.h storage/falcon/SRLUpdateRecords.cpp storage/falcon/SRLUpdateRecords.h storage/falcon/Table.h per-file messages: storage/falcon/Record.h Disable CHECK_RECORD_ACTIVITY. This flag has not been maintained, and causes invalid fatal assertions. storage/falcon/SRLUpdateRecords.cpp SRLUpdateRecords::chill() now gets an exclusive syncPrior lock before deleting the record data of a record being chilled. To prevent a deadlock with concurrent record thaws, abandon the chill for any record for which the syncPrior cannot be acquired within 50ms. The presumption is that if syncPrior is held for a record chain, then all records should remain fully intact. storage/falcon/SRLUpdateRecords.h Change return type of SRLUpdateRecords::chill() to boolean storage/falcon/Table.h Increase size of syncPrior and syncThaw hash table === modified file 'storage/falcon/Record.h' --- a/storage/falcon/Record.h 2009-01-09 22:00:52 +0000 +++ b/storage/falcon/Record.h 2009-01-17 08:22:44 +0000 @@ -27,7 +27,7 @@ #define ALLOCATE_RECORD(n) (char*) MemMgrRecordAllocate (n, __FILE__, __LINE__) #define DELETE_RECORD(record) MemMgrRecordDelete (record); -#define CHECK_RECORD_ACTIVITY +//#define CHECK_RECORD_ACTIVITY #include "SynchronizationObject.h" === modified file 'storage/falcon/SRLUpdateRecords.cpp' --- a/storage/falcon/SRLUpdateRecords.cpp 2008-11-19 17:00:02 +0000 +++ b/storage/falcon/SRLUpdateRecords.cpp 2009-01-17 08:22:44 +0000 @@ -28,6 +28,7 @@ #include "Sync.h" #include "SerialLogWindow.h" #include "Format.h" +#include "SQLError.h" SRLUpdateRecords::SRLUpdateRecords(void) { @@ -37,8 +38,19 @@ SRLUpdateRecords::~SRLUpdateRecords(void { } -void SRLUpdateRecords::chill(Transaction *transaction, RecordVersion *record, uint dataLength) +bool SRLUpdateRecords::chill(Transaction *transaction, RecordVersion *record, uint dataLength) { + Sync syncPrior(record->getSyncPrior(), "SRLUpdateRecords::chill"); + + try + { + syncPrior.lock(Exclusive, 50); + } + catch (...) + { + return false; + } + // Record data has been written to the serial log, so release the data // buffer and set the state accordingly @@ -55,6 +67,8 @@ void SRLUpdateRecords::chill(Transaction transaction->totalRecordData -= dataLength; else transaction->totalRecordData = 0; + + return true; } int SRLUpdateRecords::thaw(RecordVersion *record, bool *thawed) @@ -200,15 +214,16 @@ void SRLUpdateRecords::append(Transactio { int chillBytes = record->getEncodedSize(); - chill(transaction, record, chillBytes); - - log->chilledRecords++; - log->chilledBytes += chillBytes; + if (chill(transaction, record, chillBytes)) + { + log->chilledRecords++; + log->chilledBytes += chillBytes; - ASSERT(transaction->thawedRecords > 0); + //ASSERT(transaction->thawedRecords > 0); - if (transaction->thawedRecords) - transaction->thawedRecords--; + if (transaction->thawedRecords) + transaction->thawedRecords--; + } } else { @@ -253,9 +268,11 @@ void SRLUpdateRecords::append(Transactio if (chillRecords && record->state != recDeleted) { - chill(transaction, record, stream.totalLength); - chilledRecordsWindow++; - chilledBytesWindow += stream.totalLength; + if (chill(transaction, record, stream.totalLength)) + { + chilledRecordsWindow++; + chilledBytesWindow += stream.totalLength; + } } } // next record version === modified file 'storage/falcon/SRLUpdateRecords.h' --- a/storage/falcon/SRLUpdateRecords.h 2008-11-14 02:30:11 +0000 +++ b/storage/falcon/SRLUpdateRecords.h 2009-01-17 08:22:44 +0000 @@ -33,7 +33,7 @@ public: virtual void read(void); virtual void pass2(void); void append(Transaction *transaction, RecordVersion *records, bool chillRecords = false); - void chill(Transaction *transaction, RecordVersion *record, uint dataLength); + bool chill(Transaction *transaction, RecordVersion *record, uint dataLength); int thaw(RecordVersion *record, bool *thawed); const UCHAR *data; === modified file 'storage/falcon/Table.h' --- a/storage/falcon/Table.h 2009-01-15 20:29:54 +0000 +++ b/storage/falcon/Table.h 2009-01-17 08:22:44 +0000 @@ -42,8 +42,8 @@ static const int PostCommit = 128; static const int BL_SIZE = 128; static const int FORMAT_HASH_SIZE = 20; -static const int SYNC_VERSIONS_SIZE = 16; -static const int SYNC_THAW_SIZE = 16; +static const int SYNC_VERSIONS_SIZE = 32; // 16; +static const int SYNC_THAW_SIZE = 32; // 16; #define FOR_FIELDS(field,table) {for (Field *field=table->fields; field; field = field->next){ -- MySQL Code Commits Mailing List For list archives: http://lists.mysql.com/commits To unsubscribe: http://lists.mysql.com/commits?unsub=klewis@stripped --Boundary_(ID_7IU+CYIY9/gJL3OJz8Uweg) Content-type: message/rfc822; name="Attached Message" Return-path: Received: from fe-amer-09.sun.com ([unknown] [192.18.109.79]) by amer3-mail1.central.sun.com (Sun Java(tm) System Messaging Server 7.0-3.01 64bit (built Dec 9 2008)) with ESMTP id <0KDJ00G9DXJ1P800@stripped> for kl221630@stripped; Thu, 15 Jan 2009 23:47:25 -0700 (MST) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0KDJ00G01XDUSA00@stripped> for kl221630@stripped (ORCPT kevin.lewis@stripped); Thu, 15 Jan 2009 23:47:25 -0700 (MST) Received: from phys-amer3-2.central.sun.com ([129.147.157.45]) by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTP id <0KDJ00I99XJ198B0@stripped> for kl221630@stripped (ORCPT kevin.lewis@stripped); Thu, 15 Jan 2009 23:47:25 -0700 (MST) Received: from dm-central-01.central.sun.com ([unknown] [129.147.62.4]) by amer3-mail1.central.sun.com (Sun Java(tm) System Messaging Server 7.0-3.01 64bit (built Dec 9 2008)) with ESMTP id <0KDJ00G9BXJ1P800@stripped> for kl221630@stripped (ORCPT kevin.lewis@stripped); Thu, 15 Jan 2009 23:47:25 -0700 (MST) Received: from sunmail5.uk.sun.com (sunmail5.UK.Sun.COM [129.156.85.165]) by dm-central-01.central.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n0G6lOtw011959 for ; Thu, 15 Jan 2009 23:47:25 -0700 (MST) Received: from nwk-avmta-2.sfbay.sun.com (nwk-avmta-2.SFBay.Sun.COM [129.145.155.6]) by sunmail5.uk.sun.com (8.13.8+Sun/8.13.8/ENSMAIL,v2.2) with ESMTP id n0G6lJ4V013701 for <@sunmail2sca.sfbay.sun.com:kevin.lewis@stripped>; Fri, 16 Jan 2009 06:47:24 +0000 (GMT) Received: from pmxchannel-daemon.nwk-avmta-2.sfbay.sun.com by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) id <0KDJ00M01XJ0OJ00@stripped> for kevin.lewis@stripped (ORCPT kevin.lewis@stripped); Thu, 15 Jan 2009 22:47:24 -0800 (PST) Received: from dm-xlate-01.central.sun.com ([129.147.62.11]) by nwk-avmta-2.sfbay.sun.com (Sun Java System Messaging Server 6.2-3.04 (built Jul 15 2005)) with ESMTP id <0KDJ00F3BXIZQJ70@stripped> for kevin.lewis@stripped (ORCPT kevin.lewis@stripped); Thu, 15 Jan 2009 22:47:23 -0800 (PST) Received: from brmea-mail-2.sun.com (brmea-mail-2.Sun.COM [192.18.98.43]) by dm-xlate-01.central.sun.com (8.13.8+Sun/8.13.8) with ESMTP id n0G6lNgn011940 for ; Thu, 15 Jan 2009 23:47:23 -0700 (MST) Received: from fe-amer-10.sun.com ([192.18.109.80]) by brmea-mail-2.sun.com (8.13.6+Sun/8.12.9) with ESMTP id n0G6lMsh016528 for ; Fri, 16 Jan 2009 06:47:22 +0000 (GMT) Received: from conversion-daemon.mail-amer.sun.com by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) id <0KDJ00J01XH4JE00@stripped> (original mail from Christopher.Powers@stripped) for klewis@stripped; Thu, 15 Jan 2009 23:47:22 -0700 (MST) Received: from [10.1.10.157] (c-71-63-235-1.hsd1.mn.comcast.net [71.63.235.1]) by mail-amer.sun.com (Sun Java System Messaging Server 6.2-8.04 (built Feb 28 2007)) with ESMTPSA id <0KDJ00MN5XIYQ750@stripped>; Thu, 15 Jan 2009 23:47:22 -0700 (MST) Date: Fri, 16 Jan 2009 00:47:06 -0600 From: Christopher Powers Subject: Re: Bug#36631, "Assertion in SerialLogControl::nextRecord" In-reply-to: <49702D4B.8090104@stripped> Sender: Christopher.Powers@stripped To: Kevin Lewis Cc: FalconDev Message-id: <49702D6A.3080904@stripped> MIME-version: 1.0 Content-type: multipart/mixed; boundary="Boundary_(ID_7sGLYvXnVTmwKfsFqIdkZg)" X-Mozilla-Keys: X-PMX-Version: 5.4.1.325704 References: <49702D4B.8090104@stripped> User-Agent: Thunderbird 2.0.0.19 (Windows/20081209) Original-recipient: rfc822;kevin.lewis@stripped This is a multi-part message in MIME format. --Boundary_(ID_7sGLYvXnVTmwKfsFqIdkZg) Content-type: text/plain; format=flowed; charset=ISO-8859-1 Content-transfer-encoding: 7BIT Christopher Powers wrote: > Kevin, > > This is the reopened chill/thaw stress test bug. > > I can easily reproduce an assert in Record::getEncodedValue() that > always occurs during a scavenge/record pruning (stack attached.) > > At the time of the assert, the record state is 'recChilled', so that > explains why getEncodedValue() fails. It appears that the record 'oldie' > in Index::duplicateKey() gets chilled while it's being accessed. > > Note that hasRecord() should have thawed the record. I added a second > thaw to verify this, but the breakpoint never hit, so I think the record > is being chilled during makeKey(). > > > bool Index::duplicateKey(IndexKey *key, Record * record) > { > for (Record *oldie = record; oldie; oldie = oldie->getPriorVersion()) > if (oldie->hasRecord()) <<<< will thaw record > { > IndexKey oldKey(this); > > if (oldie->state == recChilled) > oldie->thaw(); <<<< breakpoint never hit > > makeKey (oldie, &oldKey); <<<< assertion here--chill< during > makeKey?? > > if (oldKey.isEqual(key)) > return true; > } > return false; > } > --Boundary_(ID_7sGLYvXnVTmwKfsFqIdkZg) Content-type: text/plain; name=chill_stack.txt Content-transfer-encoding: 7BIT Content-disposition: inline; filename=chill_stack.txt 24 raise() 0x00002b7ab389ae4b 23 Error::debugBreak() ./storage/falcon/Error.cpp:94 22 Error::error() ./storage/falcon/Error.cpp:71 21 Error::assertionFailed() ./storage/falcon/Error.cpp:78 20 Record::getEncodedValue() ./storage/falcon/Record.cpp:668 19 Record::getRawValue() ./storage/falcon/Record.cpp:370 18 Record::getValue() ./storage/falcon/Record.cpp:322 17 Index::makeKey() ./storage/falcon/Index.cpp:607 16 Index::duplicateKey() ./storage/falcon/Index.cpp:673 15 Index::garbageCollect() ./storage/falcon/Index.cpp:627 14 Table::garbageCollect() ./storage/falcon/Table.cpp:2134 13 RecordLeaf::pruneRecords() ./storage/falcon/RecordLeaf.cpp:159 12 RecordGroup::pruneRecords() ./storage/falcon/RecordGroup.cpp:122 11 Table::pruneRecords() ./storage/falcon/Table.cpp:1898 10 Database::pruneRecords() ./storage/falcon/Database.cpp:1848 9 Database::scavengeRecords() ./storage/falcon/Database.cpp:1798 8 Database::scavenge() ./storage/falcon/Database.cpp:1759 7 Database::scavengerThreadMain() ./storage/falcon/Database.cpp:1923 6 Database::scavengerThreadMain() ./storage/falcon/Database.cpp:1912 5 Thread::thread() ./storage/falcon/Thread.cpp:167 4 Thread::thread() ./storage/falcon/Thread.cpp:146 --Boundary_(ID_7sGLYvXnVTmwKfsFqIdkZg)-- --Boundary_(ID_7IU+CYIY9/gJL3OJz8Uweg)--