List:MySQL++« Previous MessageNext Message »
From:Kurt D. Knudsen Date:November 20 2009 8:36pm
Subject:Need help optimising INSERT queries
View as plain text  
Hi all,

 

I'm modifying a current project that lists files in a directory and
writes the file attributes, plus and MD5 hash, to a simple text file.
We're trying to port this over to MySQL to help solve some issues we
have when reading large text files, especially when it's trying to read
many of them at the same time. The problem I have is that the MySQL port
I did takes significantly longer to run than the normal flat file
writing does. In the current state, it can list a specific directory,
create the md5 hashes, and output to a text file in about 8 seconds.
When I use my MySQL version, I didn't let it finish because it already
passed the 5 minute mark and from the looks of it, it wasn't close to
finishing. Here's what I'm doing in the code and I hope someone can
point me in the right direction. This won't compile because it's missing
the needed libraries and headers from the rest of the project, as well
as some other functions that don't pertain to the main project (getting
the MD5 hash, noralizedirectory, etc). I was looking into how to use
bulk inserts, but I've never used vectors before and I don't know how to
go about using them.

 

This is the first time I've had to do something like this with MySQL,
which is why I'm so lost. I'm sure there has to be a more efficient way
to insert these rows into the database and speed this thing up a bit.

 

Thanks,

 

Kurt

 

Edit: I've sent this twice with the source code, doesn't seem to want to
take. Please see: http://pastebin.com/m58c36e42


Thread
Need help optimising INSERT queriesKurt D. Knudsen20 Nov
  • Re: Need help optimising INSERT queriesWarren Young21 Nov