List:General Discussion« Previous MessageNext Message »
From:Cabbar Duzayak Date:July 9 2005 7:01am
Subject:Storing huge amount of binary data
View as plain text  
Hi Everyone,

I will be starting a project for which I will need to store about 1
million image files all of which are about 50-100K in size. I will be
using Linux for this project. The database won't be very busy, there
will be batch image uploads 1-2 times a day, and around 100-200
concurrent users at most, most of which will be reading from the db
and writing only session information type of data, etc... And, I don't
really need transaction support (InnoDB)...

Adding this up, the image data size will be around 50-100 Gb, and I
will need to store a total of 1-2 Gb text information (1K for each
image) along with each of these images...

First of all, I heard that Mysql does not perform very well when
tablesize goes above 1 Gb. Is this a myth? Image table is not a big
deal, since I can partition/distribute it to ~ 100-200 tables, i.e. by
table_(id % 100). However, text information needs to stay in a single
table (since I need to do queries on it for information) and there
will be multiple indexes over this information.

And, as you can imagine, I am not sure if mysql can handle something
like this, and was wondering if you can provide some feedback.

So my questions are:

1. The main question is, do you guys have any experience with this
much binary and regular data? Do you think Mysql can handle this much
data in a reliable manner (without corrupting data and/or
degrading/terrible performance) ?

2. Can I implement this using regular SCSI disks with regular mysql?
Or do I have need advanced solutions such as clustered, replicated,
etc?

3. Again, as you can understand, I want to minimize the cost here. If
you don't think I can use mysql, do you think Microsoft SQL server is
good enough for this task?

Thanks...
Thread
Storing huge amount of binary dataCabbar Duzayak11 Jul
  • RE: Storing huge amount of binary dataMatt Babineau11 Jul
  • Re: Storing huge amount of binary dataPer Jessen11 Jul
    • Re: Storing huge amount of binary dataStephen Cook12 Jul
  • Re: Storing huge amount of binary dataDevananda11 Jul