List:General Discussion« Previous MessageNext Message »
From:Barry Leslie Date:July 13 2009 2:33pm
Subject:Re: Uploading large files with mySQL
View as plain text  
Hi Dan,

The problem with BLOB is that traditionally MySQL, along with a lot of other
databases, is not designed to handle them very well. The BLOB data is passed
between the client and server as if it where the same as any other data.
This results in large memory use by both the client and server as the BLOB
is buffered on both sides. The standard solution to this was to store the
BLOBs some where in a file system and then place some form of reference to
the BLOB in the database that could then be used by the client to get the
actual data. The problem of how to set such a system up and  maintain the
externally stored data was left up to the individual application designer.

The good news is that I am working on a generic solution to this problem
called the PrimeBase Media Stream engine (PBMS) that is intended to handle
exactly what you want to do. PBMS is a specialized storage engine that works
with other storage engines to store BLOB data. The actual BLOB data is
streamed to and from the PBMS engine itself and is not passed through the
MySQL server and client interface. What is stored in the actual BLOB columns
in the normal storage engine tables is a BLOB reference that can be used to
get the real BLOB data from the PBMS engine. The PBMS engine handles the
storage of the BLOB data which may be stored locally or could be stored
remotely in Amazon S3 storage for example.

For more information please have a look at our web site:
http://www.blobstreaming.org or check out my BLOG
http://bpbdev.blogspot.com.

Good luck on your project.

Barry


On 7/13/09 5:08 AM, "Daniele Development-ML" <daniele.dml@stripped>
wrote:

> Hello,
> I'm developing a web application that requires to store large files in a
> MySQL database. The files can range up to 2Gb.
> 
> In my understanding the upper limit for the SQL queries (thus insert
> queries) is 1GB. I'm trying, for the time being, to upload files of sizes
> around 65MB, but I still get some errors - including the "MySQL server has
> gone away".
> 
> I followed the discussion in other post, and I properly set all the system
> variables (max_allowed_packet_size, wait_timeout) to the maximum value, but
> the error still occurs.
> 
> Would you suggest looking at something in particular?
> 
> For the time being, just to try this, I'm inserting the file through the
> MySQL Query Browser - just not to add other possible sources of problems on
> top of that.
> 
> Thanks!
> 
> Dan

-------------------------------------------------------------------------
Barry Leslie

SNAP Innovation Softwareentwicklung GmbH
Senior Software Engineer

Tel: (001) 250 595 4228
Fax: (001) 250 595 4233
Email: Barry.Leslie@stripped
Web: www.PrimeBase.com

SNAP Innovation Softwareentwicklung GmbH, D-22765 Hamburg,
Max-Brauer-Allee 50, Germany
Amtsgericht Hamburg HRB 61066, Geschäftsführer: Ulrich Zimmer, Paul
McCullagh
-------------------------------------------------------------------------



Thread
Uploading large files with mySQLDaniele Development-ML13 Jul
  • Re: Uploading large files with mySQLJohan De Meersman13 Jul
  • Re: Uploading large files with mySQLmos13 Jul
  • Re: Uploading large files with mySQLBarry Leslie13 Jul