List:General Discussion« Previous MessageNext Message »
From:Eberhard Lisse Date:May 22 2002 5:35pm
Subject:Re: Loading massive data set from CSV
View as plain text  
In message <PPEEKJBNGLCGILHOJMMOMEBPLEAA.dan@stripped>, "Dan Harrington" write
s:

> I have an ASCII CSV or Tab Delimited file that is roughly 3.5
> gigabytes, and I want to load it into a mysql database so I can do
> some analysis.  First of all, I'm wondering, is there anything I
> should be aware of, or worried about, size-wise?

Any ideas of how many records?
 
> I know that I can't even look at the file using basic text functions
> in my Linux box like 'head' or 'split'

But, you can hack a small perl script that reads line by line flushing
output, so you can at least look at it,.
 
> There is a list of the fields in the file, so I know what my table
> should look like, but I don't want to crash the SQL server if its
> too large a file, or something else like that.

I'd doubt that MySQL would crash about a few Million records. Well,
you'd check if the disk space is there, too :-)-O

> I didn't know how big it was originally, so I was just going to use
> phpMyAdmin to load the file through a web browser....  though I
> don't know if that will work either.  Is there a size limitation to
> HTTP-POST (I assume it uses that method to upload).

Pumping 3.5 GB through an network unnecessarily is fairly silly,
actually :-)-O

el
-- 
Dr. Eberhard W. Lisse  \        /  Obstetrician & Gynaecologist (Saar)
<el@stripped> el108    *       |             Swakopmund State Hospital
Private Bag 5004        \     /     Telephone: +49 177 214 3196 (cell)
Swakopmund, Namibia     ;____/  Currently on Post Graduate Study Leave
Thread
connecting with myodbcMir S Islam13 Mar
  • connecting with myodbcsinisa14 Mar
  • Re: Pre-release of MySQL 3.23.50Philip Molter23 Apr
    • Re: Pre-release of MySQL 3.23.50(Trond Eivind Glomsrød)23 Apr
  • Re: Pre-release of MySQL 3.23.50Unknown Sender23 Apr
    • Re: Pre-release of MySQL 3.23.50Michael Widenius15 May
      • Loading massive data set from CSVDan Harrington22 May
        • Re: Loading massive data set from CSVEberhard Lisse22 May
        • Re: Loading massive data set from CSVGavin Brown22 May
  • Download of sourceLuciano Barcaro29 Apr
    • Re: Download of sourcePaul DuBois29 Apr
  • Re: Download of sourceLuciano Barcaro29 Apr
Re: Pre-release of MySQL 3.23.50David Huxtable25 Apr