List:Cluster« Previous MessageNext Message »
From:Craig L Russell Date:November 27 2012 6:48pm
Subject:Re: Problem with multithreading and ClusterJ
View as plain text  
Hi Graeme,

I'm curious to know whether you are guaranteed to create unique  
primary keys with your test. Are you sure that the lock wait timeout  
is actually an error?

I don't have much to go on here, but you might be thinking that in  
case of a duplicate, there will be no conflict because the second  
"write" to arrive will just overwrite the first. But in fact, the  
first session to write the key will hold a lock on the row (whether it  
exists or not) until the transaction commits. The second session to  
write the key will wait until the first transaction commits.

If both transactions start, and the first transaction writes key 1,  
and the second transaction writes key 2, and then the first  
transaction tries to write key 2, and the second transaction tries to  
write key 1, you have a real deadlock.

If the internal queue from which threads pull requests is generating  
unique keys, then we have some other problem that isn't obvious.

Another thing that occurs to me is that for better performance, you  
might want to try to restrict a session to insert only rows that  
belong to a specific partition. That is, calculate the partition key  
for the row and then delegate the write operation to the thread  
(session) that is already operating in that partition.

Craig

On Nov 27, 2012, at 6:37 AM, Graeme Wallace wrote:

> Magnus,
>
> Here's the schema -
>
> CREATE TABLE `linkedavailableroute2` (
>  `routeKey1` bigint(20) NOT NULL,
>  `routeKey2` bigint(20) NOT NULL,
>  `routeKey3` bigint(20) NOT NULL,
>  `pointOfSaleCountryCode` char(3) DEFAULT NULL,
>  `possibleDupeJourney` char(1) DEFAULT NULL,
>  `pointOfCommencementCityCode` char(3) DEFAULT NULL,
>  `destinationCityCode` char(3) DEFAULT NULL,
>  `airportCodes` varchar(24) DEFAULT NULL,
>  `lastModifiedDate` int(11) DEFAULT '0',
>  `lastModifiedTime` int(11) DEFAULT '0',
>  `availability` char(26) NOT NULL,
>  `travelDateOffsets` smallint(6) DEFAULT '0',
>  PRIMARY KEY (`routeKey1`,`routeKey2`,`routeKey3`),
>  KEY `lastModifiedDateIndex` (`lastModifiedDate`)
> ) ENGINE=ndbcluster DEFAULT CHARSET=latin1
>
> and the Java Interface
>
> @PersistenceCapable(table="linkedavailableroute2")
> public interface Availability {
>
>    @Column(name="routeKey1")
>    long getRouteKey1();
>    void setRouteKey1(long routeKey1);
>
>    @Column(name="routeKey2")
>    long getRouteKey2();
>    void setRouteKey2(long routeKey2);
>
>    @Column(name="routeKey3")
>    long getRouteKey3();
>    void setRouteKey3(long routeKey3);
>
>    @Column(name="pointOfSaleCountryCode")
>    String getPointOfSaleCountryCode();
>    void setPointOfSaleCountryCode(String pointOfSaleCountryCode);
>
>    @Column(name="possibleDupeJourney")
>    String getPossibleDupeJourney();
>    void setPossibleDupeJourney(String possibleDupeJourney);
>
>    @Column(name="pointOfCommencementCityCode")
>    String getPointOfCommencementCityCode();
>    void setPointOfCommencementCityCode(String  
> pointOfCommencementCityCode);
>
>    @Column(name="destinationCityCode")
>    String getDestinationCityCode();
>    void setDestinationCityCode(String destinationCityCode);
>
>    @Column(name="airportCodes")
>    String getAirportCodes();
>    void setAirportCodes(String airportCodes);
>
>    @Column(name="lastModifiedDate")
>    int getLastModifiedDate();
>    void setLastModifiedDate(int lastModifiedDate);
>
>    @Column(name="lastModifiedTime")
>    int getLastModifiedTime();
>    void setLastModifiedTime(int lastModifiedTime);
>
>    @Column(name="availability")
>    String getAvailability();
>    void setAvailability(String availability);
>
>    @Column(name="travelDateOffsets")
>    short getTravelDateOffsets();
>    void setTravelDateOffsets(short travelDateOffsets);
>
> }
>
> And the run() method from my Runnable implementation - (i have a
> ExecutorService with a configurable number of threads that pulls  
> these off
> its internal queue)
>
>       public void run() {
>
>            Session      session        =
> ThreadContext.getInstance().getSession().get();
>            Availability availability   =
> ThreadContext.getInstance().getAvailability().get();
>            Integer      persistedCount =
> ThreadContext.getInstance().getPersistedCount().get();
>
>            if (session == null) {
>                System.out.println("Thread " +
> Thread.currentThread().getName() + " creating session");
>                session =
> ThreadContext.getInstance().getSessionFactory().getSession();
>                session.currentTransaction().begin();
>
>                ThreadContext.getInstance().getSession().set(session);
>
>                availability = session.newInstance(Availability.class);
>
>
> ThreadContext.getInstance().getAvailability().set(availability);
>
>                persistedCount = 0;
>            }
>
>
> availability 
> .setRouteKey1(AvailabilityHelper.generateRouteKey(_originCityCode,
> _destinationCityCode, _carrierCode, _travelDate));
>            availability.setRouteKey2(_flightNumber);
>            availability.setRouteKey3(0);
>
>            availability.setAirportCodes(_airportCodes);
>            availability.setAvailability(_availability);
>            availability.setDestinationCityCode(_destinationCityCode);
>            availability.setLastModifiedDate(_lastModifiedDate);
>            availability.setLastModifiedTime(_lastModifiedTime);
>             
> availability.setPointOfSaleCountryCode(_pointOfSaleCountryCode);
>            availability.setPossibleDupeJourney(_possibleDupeJourney);
>
> availability 
> .setPointOfCommencementCityCode(_pointOfCommencementCityCode);
>            availability.setTravelDateOffsets(_travelDateOffsets);
>
>            session.savePersistent(availability);
>
>
> ThreadContext.getInstance().getPersistedCount().set(++persistedCount);
>
>            if (persistedCount % TRANSACTION_LIMIT == 0) {
>                System.out.println("Thread " +
> Thread.currentThread().getName() + " committing " + persistedCount);
>                session.currentTransaction().commit();
>                session.currentTransaction().begin();
>            }
>
>        }
>
> The ThreadContext is just a singleton wrapper around some  
> ThreadLocal's to
> store Session and Availability objects. SessionFactory is an instance
> variable.
>
>
>
> On Tue, Nov 27, 2012 at 6:12 AM, Magnus Blåudd <magnus.blaudd@stripped 
> >wrote:
>
>> Hi Graeme,
>>
>> Normally MySQL Cluster is able to cope with very high read and  
>> write load,
>> but unfortunately something is not right. Do you think you could  
>> show us
>> schema and the code you use?
>>
>> Quickly looked in SaveTest.java which shows how to use  
>> savePersistent()
>> and savePersistentAll(), I assume you use the latter? Could serve  
>> as a
>> starting point for a reproducable public test case.
>>
>>
>> / Magnus
>>
>>
>> On 11/26/2012 06:49 PM, Graeme Wallace wrote:
>>
>>> Ok tried reducing transaction size down to 4K rows. Works for 2  
>>> threads,
>>> but its slower than doing one thread with larger transactions.
>>>
>>> If i up the threads to 4, i get the same error as before
>>>
>>> Nov 26, 2012 12:44:24 PM com.mysql.clusterj.tie.Utility throwError
>>> SEVERE: Error in NdbJTie: returnCode -1, code 266, mysqlCode 146,  
>>> status
>>> 1, classification 10, message Time-out in NDB, probably caused by
>>> deadlock .
>>>
>>>
>>> Am I just going about this wrong ? I assumed that as i have a  
>>> cluster of
>>> 10 datanodes i would be able to have lots of threads and therefore  
>>> lots
>>> of writes going on at the same time. (Caveat - i read the marketing
>>> material for 1 billion reads and writes a second :) )
>>>
>>> regards,
>>>
>>>
>>>
>>> Graeme
>>>
>>>
>>>
>>> On Mon, Nov 26, 2012 at 8:44 AM, Magnus Blåudd
> <magnus.blaudd@stripped
>>> <mailto:magnus.blaudd@oracle.**com
> <magnus.blaudd@stripped>>>  
>>> wrote:
>>>
>>>    On Mon 26 Nov 2012 04:33:57 PM CET, Graeme Wallace wrote:
>>>
>>>        I've tried playing around with transaction size, but the  
>>> error
>>> still
>>>        remains even if I make the transaction small ie 32K rows  
>>> AND i
>>>        make all the
>>>        primary keys unique - so that each transaction should have a
>>>        unique set of
>>>        keys.
>>>
>>>        G.
>>>
>>>
>>>        On Mon, Nov 26, 2012 at 8:30 AM, Magnus Blåudd
>>>        <magnus.blaudd@stripped  
>>> <mailto:magnus.blaudd@oracle.**com<magnus.blaudd@stripped>
>>>>> __wrote:
>>>
>>>            On Mon 26 Nov 2012 04:09:42 PM CET, Graeme Wallace wrote:
>>>
>>>                I'm trying to make my app multi-threaded and  
>>> running into
>>>                problems.
>>>
>>>                I have a Session local to each thread that is  
>>> attempting
>>>                to write to the
>>>                db. I'm batching up many savePersistent() calls in a
>>>                transaction - but
>>>                when
>>>                the transaction commits I invariably end up with
>>>
>>>                Nov 21, 2012 7:22:29 PM  
>>> com.mysql.clusterj.tie.Utility
>>>                throwError
>>>                SEVERE: Error in NdbJTie: returnCode -1, code 266,
>>>                mysqlCode 146, status
>>>                1,
>>>                classification 10, message Time-out in NDB, probably
>>>                caused by deadlock .
>>>
>>>                For reading the docs, it looks like there shouldn't  
>>> be
>>>                table locking going
>>>                on - so i dont understand what resource is being  
>>> held by
>>>                one thread that
>>>                stops the others from being able to write at the same
>>> time.
>>>
>>>                Any clues would be most helpful,
>>>
>>>                regards,
>>>
>>>
>>>                Graeme
>>>
>>>
>>>            Check out Matthew's reply in in this forum thread
>>>           
> http://forums.mysql.com/read.***__*php?25,505712,505757<http://forums.mysql.com/read.*__*php?25,505712,505757
> 
>>> >
>>>           
> <http://forums.mysql.com/read.****php?25,505712,505757<http://forums.mysql.com/read.**php?25,505712,505757
> 
>>> >
>>>>
> <http:/**/__forums.mysql.com/read.php?**25,__505712,505757<http://forums.mysql.com/read.php?25,__505712,505757
> 
>>>> >
>>>           
> <http://forums.mysql.com/read.**php?25,505712,505757<http://forums.mysql.com/read.php?25,505712,505757
> 
>>> >
>>>>>
>>>
>>>            "Due to the distributed nature of NDBCLUSTER there is no
>>>            automatic
>>>            deadlock detection mechanism within the NDBCLUSTER  
>>> engine.
>>>            The only
>>>            indication that the cluster gives that there is a  
>>> *possible*
>>>            deadlock is
>>>            the "Lock wait timeout exceeded". This error occurs  
>>> when a
>>>            given lock
>>>            operation takes longer than
>>>            TransactionDeadlockDetectionTi**__**meout
>>>            (default 1200 ms.). If you have large transactions that  
>>> lock
>>>            many rows at
>>>            once or if you have long running transactions these can
>>>            prevent this or
>>>            transactions from completing quickly. Try to avoid  
>>> large or
>>>            long running
>>>            transactions by breaking them into smaller chunks. The
>>>            TransactionDeadlockDetectionTi**__**meout can also be  
>>> reached
>>>            when a node is
>>>            overloaded but has not yet been ejected from the  
>>> cluster for
>>>            missing
>>>            heartbeats longer than 4xHeartbeatIntervalDbDb (default  
>>> 1500
>>>            ms. each). "
>>>
>>>            / Magnus
>>>
>>>
>>>
>>>
>>>
>>>    Give it a try with even smaller transactions if possible.  
>>> Normally
>>>    better to use more threads with smaller transactions.
>>>
>>>    / Magnus
>>>
>>>
>>>
>>>
>>> --
>>> Graeme Wallace
>>> CTO
>>> FareCompare.com
>>> O: 972 588 1414
>>> M: 214 681 9018
>>>
>>>
>>>
>>
>
>
> -- 
> Graeme Wallace
> CTO
> FareCompare.com
> O: 972 588 1414
> M: 214 681 9018

Craig L Russell
Architect, Oracle
http://db.apache.org/jdo
408 276-5638 mailto:Craig.Russell@stripped
P.S. A good JDO? O, Gasp!

Thread
Problem with multithreading and ClusterJGraeme Wallace26 Nov
  • Re: Problem with multithreading and ClusterJMagnus Blåudd26 Nov
    • Re: Problem with multithreading and ClusterJGraeme Wallace26 Nov
      • Re: Problem with multithreading and ClusterJMagnus Blåudd26 Nov
        • Re: Problem with multithreading and ClusterJGraeme Wallace26 Nov
          • Re: Problem with multithreading and ClusterJMagnus Blåudd27 Nov
            • Re: Problem with multithreading and ClusterJGraeme Wallace27 Nov
              • Re: Problem with multithreading and ClusterJCraig L Russell27 Nov