Re: question about chunking sql uploads

[Scott Gifford]

SGreen@stripped writes:

> Frank Bax <fbax@stripped> wrote on 04/25/2005 11:47:12 AM:
>
>> At 10:44 AM 4/25/05, Art.M (Wikki) wrote:
>> >I have a large .sql file to upload which is about 9 mb and I was
>> >wondering if anyone knew of a program that could break it up into
>> >chunks of 2 mb or under? So I can upload it to a shared web server.
>> 
>> 
>> You can't upload a 9M file to webserver?  But you can upload 9x1M files? 
>
>> Fix your webserver.  You didn't google for this did you?  You also
>> didn't mention OS, so we'll assume unix based.

This is probably more common than you would expect.  I've run into it
when using shared hosting providers; they have timeouts and resource
limits on programs like phpMyAdmin, and so can't handle big chunks of
data.  It's not very easy to call up a large shared hosting provider
and tell them "fix your server", especially if you are a small
customer.

That said, I don't have a great solution.  I often "re-chunk" these
files by hand, which isn't that bad if you only need to break it into
5-10 chunks.  It would be straightforward to write a Perl script to do
this.

>> You could try compressing file with zip, gzip, etc.

This often helps, too, if the program you're importing data with
supports it.

>> Or simply use "split", default is 1000 lines, but can be changed via 
>> command line.

That's a start, but the files each need to be a valid SQL statement,
so that's not enough; a little fixup needs to happen at the beginning
and end of each file.

---ScottG.