List:General Discussion« Previous MessageNext Message »
From:Vikram A Date:February 24 2010 4:15am
Subject:Re: how things get messed up
View as plain text  
Sirs,

Because one table will hold the large amount of data, only the recent data will be used
for transactions; so rest of the old records are remain same with out any transaction. So
we have decided to go for year based storage; here even old records can be taken out by
join queries. 

I hope you experts will agree with this. Or your comments and suggestions are welcome 
for the better design. 

Thank you

VIKRAM A




________________________________
From: Jerry Schwartz <jschwartz@stripped>
To: Vikram A <vikkiatbipl@stripped>
Cc: Johan De Meersman <vegivamp@stripped>; MY SQL Mailing list
<mysql@stripped>
Sent: Tue, 23 February, 2010 3:53:38 AM
Subject: RE: how things get messed up

 
I thought I had replied publicly to Johan’s suggestion, with
some personal experience.
 
He’s absolutely right, that would give you a solution that would
be completely transparent to your application and therefore much easier to
implement. You could keep re-arranging your partitions as necessary.
 
I, myself, have never used portioning so I hope someone with
experience will chime in here.
 
One disadvantage is that all of your data would be in one
database, making your backups bigger and bigger. If you used a separate
database as an archive, the archive database wouldn’t have to be backed up very
often.
 
I never did get a good feel for how big your database will be.
Even if you’re talking about IIT, and assuming 100000 students, 6 classes per
semester, three semesters, per year, 20 years of history, you’re going to have 36
million class records. I think there are much bigger databases running quite
well.
 
Regards,
 
Jerry Schwartz
The Infoshop by Global Information Incorporated
195 Farmington Ave.
Farmington, CT 06032
 
860.674.8796 / FAX: 860.674.8341
 
www.the-infoshop.com
 
From:Vikram A
[mailto:vikkiatbipl@stripped] 
Sent: Friday, February 19, 2010 11:17 PM
To: Jerry Schwartz
Cc: Johan De Meersman
Subject: Re: how things get messed up
 
Dear
Sir,

I agree with the solution proposed. 

But one of the member[Johan De Meersman<vegivamp@stripped>] of this list has
commented it.

Do you have any opposition/Suggestions?

Thank you

VIKRAM A

________________________________
 
From:Jerry Schwartz
<jschwartz@stripped>
To: Vikram A <vikkiatbipl@stripped>
Cc: MY SQL Mailing list <mysql@stripped>
Sent: Thu, 18 February, 2010 9:54:57 PM
Subject: RE: how things get messed up
From:Vikram A
[mailto:vikkiatbipl@stripped] 
Sent: Wednesday, February 17, 2010 11:41 PM
To: Jerry Schwartz
Cc: MY SQL Mailing list
Subject: Re: how things get messed up
 
Dear Jerry Schwartz

We have applications for colleges in India. The same idea of having single
table for manipulating students records. but we are not following archiving
concept.

Ex stupersonal. and stuclass these tables are playing wide role
in our application. After 7 years now there are 9000 records[postgresql
backend] are there in the table. Because of this the entire application [ Fees,
attendance, exams etc] performance is getting down. For the remedy of this I
proposed this year wise architecture for our new version [mysql].
[JS] You have 9000 records? That should not slow down any
application. I must not understand you.

I have problem in year wise also, i have number of mutual related tables for
students such as stu_last_studies, stu_family_details, stu_address,
stu_extracurri and so on. If i go for year basisis i have to make all the above
tables also year basis. 
Hence, I feel it difficult have such number of tables after few years. 
[JS] I did not mean that you should have tables for each year. I
was suggesting that you have tables for recent data and tables for archived
data. 

As you said the archive system, can you the idea about the archive system[If
needed i will give the table structures]. 
[JS] This is best described with a picture. Here is a small
example of what I meant:
 
         
`student_master_table`  (all years)
                   
/\
                  
/  \
     `grades_current`  `grades_archive`
               
|         /
     `class_master_table`
 
The structures of the two grades tables should be almost the
same, something like
 
grade_id <autoincrement in grades_current only>
student_id <index>
class_id <index>
class_start_date
grade_received
 
You would add new grade records to the `grades_current` table.
 
Now, suppose that you don’t usually need data more than five
years old. Once a year you would run these queries:
 
INSERT INTO `grades_archive` SELECT * FROM `grades_current` WHERE
`class_start_date` < YEAR(DATE_SUB(NOW(), INTERVAL 4 YEAR));
DELETE FROM `grades_current` WHERE `class_start_date` <
YEAR(DATE_SUB(NOW(), INTERVAL 4 YEAR));
 
That would keep the `grades_current` table small. If you want to
find a student’s recent grade history, you would use a query like
 
SELECT * FROM `grades_current` WHERE `student_id` = 12345;
 
If you decide that you need a student’s complete history, you
could do 
 
SELECT * FROM `grades_current` WHERE `student_id` = 12345 UNION
ALL SELECT * FROM `grades_archive` WHERE `student_id` = 12345;
 
That is a quick outline of what I was saying.
 
I don’t know how big your database is, so I can’t begin to guess
whether or not this is necessary. On my desktop computer, where I do my
testing, I have two tables: one has about 104000 records, the other has about
200000 records. The query 
 
SELECT `prod`.`prod_num`, `prod_price`.`prod_price_del_format`,
`prod_price`.`prod_price_end_price` FROM `prod` JOIN `prod_price` ON
`prod`.`prod_id` = `prod_price`.`prod_id` WHERE `prod`.`prod_num` = 40967;
 
took .70 seconds. Repeating the same query with different values
of `prod_num` gave increasingly faster results, showing that caching is working
as expected: after three such queries, the response time was .14 seconds.
 
I understand that schools in India can be very, very big; so
perhaps you need an archive scheme such as the one I described. In fact, it
might be useful to extend this whole concept to using an archive database,
rather than archive tables within the same database. The database engine
wouldn’t really care, but since the archive database wouldn’t change very
often
you wouldn’t have to back it up very often, either.
 
Regards,
 
Jerry Schwartz
The Infoshop by Global Information Incorporated
195 Farmington Ave.
Farmington, CT 06032
 
860.674.8796 / FAX: 860.674.8341
 
www.the-infoshop.com
 
 

It will be grate help to me.

Thank you

VIKRAM A
 

________________________________
 
From:Jerry Schwartz
<jschwartz@stripped>
To: Vikram A <vikkiatbipl@stripped>; Johan De Meersman
<vegivamp@stripped>
Cc: MY SQL Mailing list <mysql@stripped>
Sent: Tue, 16 February, 2010 9:32:22 PM
Subject: RE: how things get messed up

>-----Original Message-----
>From: Vikram A [mailto:vikkiatbipl@stripped]
>Sent: Friday, February 12, 2010 4:13 AM
>To: Johan De Meersman
>Cc: MY SQL Mailing list
>Subject: Re: how things get messed up
>
>Sir,
>
>Thanks for your suggestion,
>I will go for blob storage, because our application will maintain the data
on
>yearly basis[stupersonal2008, stupersonal2009 etc.]. So i feel we may not 
>face
>such kind of performance issue in our application.
>
[JS] It sounds like you are planning to have one table per year. Regardless of 
where you put your blobs, I think that is a bad idea from a design standpoint. 
It will make it harder to find historical information.

If your database is relatively small, then I'd just keep everything in one 
table. If it is big, then roll data that is five years old into an archive 
table. That will give you only two places, and an easy-to-follow rule to tell 
you where to look.

Regards,

Jerry Schwartz
The Infoshop by Global Information Incorporated
195 Farmington Ave.
Farmington, CT 06032

860.674.8796 / FAX: 860.674.8341

www.the-infoshop.com





-- 
MySQL General Mailing List
For list archives: http://lists.mysql.com/mysql
To unsubscribe:    http://lists.mysql.com/mysql?unsub=1
 

________________________________
 
Your Mail works best with the New Yahoo Optimized IE8. Get it NOW!.



________________________________
 
The INTERNET now has a personality. YOURS! See
your Yahoo! Homepage.


      Your Mail works best with the New Yahoo Optimized IE8. Get it NOW!
http://downloads.yahoo.com/in/internetexplorer/
Thread
how things get messed upJohn G. Heim10 Feb
  • Re: how things get messed upBuford Tannen10 Feb
  • RE: how things get messed upIlya Kazakevich10 Feb
    • Re: how things get messed upCui Shijun11 Feb
      • RE: how things get messed upJerry Schwartz11 Feb
        • Re: how things get messed upJohan De Meersman12 Feb
    • Re: how things get messed upJoerg Bruehe11 Feb
    • Re: how things get messed upJochem van Dieten11 Feb
  • Re: how things get messed upMartijn Tonies11 Feb
    • Re: how things get messed upDavid Giragosian11 Feb
  • RE: how things get messed upJerry Schwartz11 Feb
  • Re: how things get messed upMartijn Tonies11 Feb
    • RE: how things get messed upJerry Schwartz11 Feb
    • Re: how things get messed upAnn W. Harrison12 Feb
  • Re: how things get messed upMartijn Tonies12 Feb
    • Re: how things get messed upJohan De Meersman12 Feb
  • Re: how things get messed upMartijn Tonies12 Feb
    • Re: how things get messed upJohan De Meersman12 Feb
      • Re: how things get messed upVikram A12 Feb
        • Re: how things get messed upJohan De Meersman12 Feb
          • Re: how things get messed upVikram A12 Feb
            • RE: how things get messed upJerry Schwartz16 Feb
              • RE: how things get messed upMartin Gainty16 Feb
                • Re: how things get messed upBarry Leslie17 Feb
              • Re: how things get messed upVikram A18 Feb
                • RE: how things get messed upJerry Schwartz18 Feb
                  • Re: how things get messed upJohan De Meersman18 Feb
  • Re: how things get messed upMartijn Tonies15 Feb
  • Re: how things get messed upMartijn Tonies15 Feb
    • Re: how things get messed upJohan De Meersman16 Feb
      • Re: how things get messed upPaul McCullagh17 Feb
  • Re: how things get messed upMartijn Tonies15 Feb
    • Re: how things get messed upAnn W. Harrison15 Feb
      • Re: how things get messed upJohan De Meersman16 Feb
      • RE: how things get messed upJerry Schwartz16 Feb
  • Re: how things get messed upMartijn Tonies18 Feb
RE: how things get messed upJerry Schwartz22 Feb
  • Re: how things get messed upVikram A24 Feb