List:General Discussion« Previous MessageNext Message »
From:Martin Ramsch Date:September 14 1999 12:56am
Subject:Re: Newbie: Need help building index table
View as plain text  
On Mon, 1999-09-13 19:58:33 -0500, Chip Castle wrote:
> I have written a perl script that builds the domain_index table by
> iterating thru EACH domain name in the domain table, producing all of
> the possible search patterns, and then inserts these patterns and domain
> reference into the domain_index table.  At this point, the script has
> been running all day and the domain_index table only has 1M records in
> it so far.  I'm expecting it to have 50-100M records when it's done, so
> it's going to take forever and I don't have that long.  Maybe my
> approach is all wrong, but I'm stuck and don't have any other ideas.

I guess your perl script does single INSERTs step by step?  You can
speed up the insert process by collecting the data (approx. 1000 items
is a good number) and then inserting all these rows at once in a
single INSERT statement.
But this still could be to slow.
Drop the index from domain_index until you're done with all inserts,
and only then add the index again.  This should dramatically speed up
the insertions, and though adding the index afterwards will need quite
long a time, this should be the fastest way.

Martin Ramsch <m.ramsch@stripped> <URL: >
PGP KeyID=0xE8EF4F75 FiPr=52 44 5E F3 B0 B1 38 26  E4 EC 80 58 7B 31 3A D7
Newbie: Need help building index tableChip Castle14 Sep
  • Re: Newbie: Need help building index tableMartin Ramsch14 Sep