List:General Discussion« Previous MessageNext Message »
From:Jay J Date:December 6 1999 4:00pm
Subject:Re: Memory leak using Perl5/DBI.pm on Mysql version: 3.22.25
View as plain text  
----- Original Message -----
From: "Jim Allen" <black@stripped>
To: "Sasha Pachev" <sasha@stripped>
Cc: <mysql@stripped>
Sent: Monday, December 06, 1999 12:39 PM
Subject: Re: Memory leak using Perl5/DBI.pm on Mysql version: 3.22.25


> I've had memory leek problems with the DBI before too. This one for
> example...
>
> #############
> use DBI;
>
> $dbh = DBI->connect("dbi:mysql:$database;hostname=$host");
> $sth = $dbh->prepare($query);
> $sth->prepare();
>
> while ($record = $sth->fetchrow_hashref) {
>
> }
> ##############
>
> If the query responds with a couple thousand entries... I get an "out of
> memory" message from the os. The exact same query, with an array being
> passed in the while loop, as opposed to a hash will parse several
> hundred thousand records without even putting a dent in the memory.
>
> What I suspect is that either DBI.pm or Mysql.pm is leaving references
> to hashes it creates lying around. If you're using fatchrow_hashref and
> getting memory leeks, sticking with a plain array from the fetchrow
> method.
>
> Jim...

Howdy,

I know there have* been leaks in DBI/DBD's but I don't think that's the
problem with the above code.

I (somewhat foolishly) stumbled onto the difference between use_result and
store_result this way, on the mod_perl list. Wondering, why do I have 20MB
servers when using hashref?? I was selecting from a 50MB database using NOT
LIKE "%foo%".

As it turned out, it had nothing to do with MySQL or DBD::mysql but rather
that I needed to use: my $sth = $dbh->prepare($sql, {mysql_use_result =>
1}); ... if I wanted to avoid slurping all the rows into memory.

So, when you say a couple thousand rows ... I'm betting it's the way it's
the # of rows, not the way they are fetched.

-Jay J


> Sasha Pachev wrote:
> >
> > Xeno Campanoli wrote:
> > >
> > > I've got a memory leak with a perl5 / DBI.pm loading script using
Mysql version: 3.22.25.  I make lots of
> > > objects, but Perl is supposed to be pretty good about garbage
collection and I cannot find any obvious cases
> > > of recursive addressing in my Perl5 code, which is allegedly
(according to the camel book) the only way to
> > > get a memory leak in Perl5.  I'm looking for suggestions or methods
for finding such a leak.  I start out with
> > > most RAM and all of swap unused (RedHat Linux) and by the end all of
the RAM and most or all of the swap
> > > is used up.  The script takes data from an older version of Mysql and
uses a Mysql.pm wrapper to read the
> > > old one, then makes objects using plain DBI.pm to write the new.  The
same DBI.pm is used for Mysql.pm.
> > > The old database is version 3.22.13-beta.  We are trying to get off of
this.  The script is not running on the
> > > system with the beta Mysql.  Please respond with suggestions.
Sincerely, Xeno
> > >
> > >
> >
> > One problem with languages that handle memory management themselves
> > (Java, Perl) as opposed to the ones where the programmer is responsible
> > for doing it, is that often the programmer has too much faith in the
> > language's ability to recognize that the memory should be freed. In any
> > of those languages, the memory for a given object cannot be freed while
> > you have a reference to it within scope. You can create a memory leak by
> > hogging up references to large objects.
> >
> > Another possibility is not using an object API properly, such as not
> > calling $sth->finish in DBI.
> >
> > --
> > Sasha Pachev
> > http://www.sashanet.com
> >
> > ---------------------------------------------------------------------
> > Please check "http://www.mysql.com/Manual_chapter/manual_toc.html"
before
> > posting. To request this thread, e-mail
mysql-thread20261@stripped
> >
> > To unsubscribe, send a message to the address shown in the
> > List-Unsubscribe header of this message. If you cannot see it,
> > e-mail mysql-unsubscribe@stripped instead.
>
> ---------------------------------------------------------------------
> Please check "http://www.mysql.com/Manual_chapter/manual_toc.html" before
> posting. To request this thread, e-mail mysql-thread20382@stripped
>
> To unsubscribe, send a message to the address shown in the
> List-Unsubscribe header of this message. If you cannot see it,
> e-mail mysql-unsubscribe@stripped instead.
>

Thread
Memory leak using Perl5/DBI.pm on Mysql version: 3.22.25Xeno Campanoli4 Dec
  • Re: Memory leak using Perl5/DBI.pm on Mysql version: 3.22.25Sasha Pachev4 Dec
  • Re: Memory leak using Perl5/DBI.pm on Mysql version: 3.22.25Jim Allen6 Dec
  • Re: Memory leak using Perl5/DBI.pm on Mysql version: 3.22.25Jay J6 Dec
  • Re: Memory leak using Perl5/DBI.pm on Mysql version: 3.22.25Xeno6 Dec