List:General Discussion« Previous MessageNext Message »
From:walt Date:May 17 2002 4:47pm
Subject:Re: Real-time data warehousing
View as plain text  
Brad Teale wrote:

> We are warehousing real-time data.  The data is received at up to T1 speeds,
> and is broken up and stored into the database in approximately 25 different
> tables.  Currently MySQL is doing terrific, we are using MyISAM tables and
> are storing 24 hours worth of data but we don't have any users and we need
> to store 72 hours worth of data.
>
> Our concern is that when we start letting our users (up to 200 simultaneous)
> hit the database, we won't be able to keep up with ingesting and serving
> data with the MyISAM locking scheme.
>
> We have tested Oracle and PostgreSQL which fell behind on the ingest.  The
> current production system uses regular ISAM files, but we need to make a
> certification which requires a relational database.  Also, the current
> production system doesn't have the feature list the new system has.
>
> Is there a better database solution or do you think MySQL can handle it?
> If MySQL can handle it, would we be better off using InnoDB or MyISAM
> tables?
>
> Thanks,
> Brad
>
> ---------------------------------------------------------------------
> Before posting, please check:
>    http://www.mysql.com/manual.php   (the manual)
>    http://lists.mysql.com/           (the list archive)
>
> To request this thread, e-mail <mysql-thread109375@stripped>
> To unsubscribe, e-mail <mysql-unsubscribe-kernel=nea-fast.com@stripped>
> Trouble unsubscribing? Try: http://lists.mysql.com/php/unsubscribe.php

Brad,
 We're in process of  evaluating mysql vs our current Oracle 8 system.  Importing
data is much faster
in mysql than oracle according the numbers we're getting. However, from our
benchmarking, Oracle seems to be faster on the queries (no writes to db during
query time). The table were running our queries against
has 46 coulmns and 14 indexes (some columns indexed twice in multi-column
indexes). All queries are based on indexed columns. We've also run into some
issues trying to delete indexes, 14+ hours before we killed the db and reloaded
data, but I may be something stupid.

One note on Oracle, $30,000+ for a single processor licence. From our testing, it
looks like the bottleneck is disk I/O not processing power.  With Oracle, you
have better control over which disks your data resides on which lets you balance
disk I/O better.  However, for $30k, you can buy 10  15,000 rpm drives, stripe
them, and then buy another server for replication of data and still have $25K
left over.

Thread
Real-time data warehousingBrad Teale17 May
  • Re: Real-time data warehousingGelu Gogancea17 May
  • Re: Real-time data warehousingwalt17 May
  • Re: Real-time data warehousingDan Nelson17 May
RE: Real-time data warehousingBrad Teale17 May
  • Re: Real-time data warehousingwalt17 May
RE: Real-time data warehousingBrad Teale17 May