We have been running our production system for 2 months on top of MySQL
3.22.13 on FreeBSD with little problems. Then it all hit the fan this
morning. We were down for 6 hours while trying to figure out why MySQL had
begun to take a very long time to process requests. Even database
independent queries like "show processlist" were taking minutes. Indexed
queries that usually take a millisecond were taking two minutes.
Things that we tried that didn't help:
ISAMCHK -r all tables
rebooting the machine
Finally we began to delete all the data from our five biggest BLOB based
tables that wasn't critical. This seemed to solve the problem, but we're
guessing as those tables begin to fill up again, we are back in trouble.
These tables are very simple (one index, a large blob) but pretty big. The
largest has about 40,000 records which average about 9K a piece (360 MBs).
We copied that database off before deleting the tables and are currently
reproducing all the work our servers were doing during the time database
freaked this morning. We can't reproduce the problem.
Anyone seen anything like this before?