List:General Discussion« Previous MessageNext Message »
From:Allen Fowler Date:September 8 2009 12:18am
Subject:Queue / FIFO in MySQL?
View as plain text  

I need to create a system where records are generated by a "producer" process and
processed by several "worker" processes.

I was thinking about something like:

1) Producer INSERTs new records with "state" = "new" & "worker" = "null"
2) Producer sleeps and loops back to step #1

1) Worker UPDATEs all records with "worker" = "pid" & "state" = "working" where
"state" == "new"
2) Worker SELECTs all records where "worker" = "pid" & "state" = "working"
3) For each record that is done, worker updates record with "state" = "done"
4) Worker loops back to step #1

Note: In this scheme the worker winds up with all "new" records generated since the last
worker claimed any. Not sure how else to guarantee atomicity. I would prefer "only n
records per request". Ideas?

I am sure something like this must have been before....  Can anyone point me to example
code, libraries, and/or refinements on the scheme?  (preferably using python...)

Thank you,

Queue / FIFO in MySQL?Allen Fowler8 Sep
  • Re: Queue / FIFO in MySQL?Hank8 Sep
  • Re: Queue / FIFO in MySQL?Perrin Harkins8 Sep
    • Re: Queue / FIFO in MySQL?Alex Arul Lurthu8 Sep
  • RE: Queue / FIFO in MySQL?Gavin Towey8 Sep
    • RE: Queue / FIFO in MySQL?Jerry Schwartz8 Sep