Re: [Boston.pm] LARGE MySQL Database

You might considering partitioning the table dependent on one of the
column's values.  The canonical example is splitting a sales table by date
range.  This has the effect of decreasing load time and query time by making
the indexes smaller.  You would have to make a series of  rules though if
you want to be able to use the partitions transparently.  I am working with
a 36+ billion row table in Postgres that is composed of 12 3 billion row
partitions, and wouldn't really be able to use it effectively without a
partitioning scheme.

-Allen



On 5/8/06, Alex Brelsfoard 
<alex.brelsfoard-Re5JQEeQqe8AvxtiuMwx3w@xxxxxxxxxxxxxxxx> wrote:
>
> Howdy all,
>
>     I know this isn't specifically a Perl question.  But I AM using Perl
> with this project.
>
> Basically I am dealing with using, storing, and sorting a LOT of data in a
> mysql database.
> With all the data in the table it makes for 404.8 Million rows.  In a
> backup
> sql file that makes just under 80GB.
>
> I an using the InnoDB engine.
>
> I was just wondering if anyone else has had experience working with
> databases this large, and using MySQL.
> I've run into some smaller problems along the way due to the immensity of
> this table.
> In the end, to do what we want I will be creating a smaller table, with a
> subset of entries from the original.
> But the original needs to exist as well.
>
> I'm looking for heads up warning for things I should watch out for due to
> the size of this thing, or any suggestions on speedier sorting and
> querying.
>
> Thanks a lot.
>
> --Alex
>
> _______________________________________________
> Boston-pm mailing list
> Boston-pm-PqP1ghmmPMdAfugRpC6u6w@xxxxxxxxxxxxxxxx
> http://mail.pm.org/mailman/listinfo/boston-pm
>