List:General Discussion« Previous MessageNext Message »
From:Matt Babineau Date:July 11 2005 9:42am
Subject:RE: Storing huge amount of binary data
View as plain text  
Sometimes, the easiest way to do this is to use the file system of the linux
machine to store the files, and make reference to them in the DB...storing
not data in the DB and getting rid of all your possible problems. 



Thanks,

Matt Babineau
Criticalcode
858.733.0160
matt@stripped
http://www.criticalcode.com
 
-----Original Message-----
From: Cabbar Duzayak [mailto:cabbar@stripped] 
Sent: Saturday, July 09, 2005 12:01 AM
To: mysql@stripped
Subject: Storing huge amount of binary data

Hi Everyone,

I will be starting a project for which I will need to store about 1 million
image files all of which are about 50-100K in size. I will be using Linux
for this project. The database won't be very busy, there will be batch image
uploads 1-2 times a day, and around 100-200 concurrent users at most, most
of which will be reading from the db and writing only session information
type of data, etc... And, I don't really need transaction support
(InnoDB)...

Adding this up, the image data size will be around 50-100 Gb, and I will
need to store a total of 1-2 Gb text information (1K for each
image) along with each of these images...

First of all, I heard that Mysql does not perform very well when tablesize
goes above 1 Gb. Is this a myth? Image table is not a big deal, since I can
partition/distribute it to ~ 100-200 tables, i.e. by table_(id % 100).
However, text information needs to stay in a single table (since I need to
do queries on it for information) and there will be multiple indexes over
this information.

And, as you can imagine, I am not sure if mysql can handle something like
this, and was wondering if you can provide some feedback.

So my questions are:

1. The main question is, do you guys have any experience with this much
binary and regular data? Do you think Mysql can handle this much data in a
reliable manner (without corrupting data and/or degrading/terrible
performance) ?

2. Can I implement this using regular SCSI disks with regular mysql?
Or do I have need advanced solutions such as clustered, replicated, etc?

3. Again, as you can understand, I want to minimize the cost here. If you
don't think I can use mysql, do you think Microsoft SQL server is good
enough for this task?

Thanks...

--
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe:    http://lists.mysql.com/mysql?unsub=1


Thread
Storing huge amount of binary dataCabbar Duzayak11 Jul
  • RE: Storing huge amount of binary dataMatt Babineau11 Jul
  • Re: Storing huge amount of binary dataPer Jessen11 Jul
    • Re: Storing huge amount of binary dataStephen Cook12 Jul
  • Re: Storing huge amount of binary dataDevananda11 Jul