I am currently running a 2.4ghz system with dual 40 Gig hard drives
and 1 GB of Ram. I do an enormous amount of ad hoc reporting and once
I get organized I will be looking to begin formalizing some of my
reporting and analysis. Currently I use Access 2000 and have VB and
Foxpro on my box, though I do not currently use them.
I have very large (some have 6 million records or more) text files
that I begin with. I then currently import them into Access 2000 and
run my queries and the like from there. (Linking to the text files is
near impossible.....takes forever to access the data.)
Here are the size and number of records of just a few of the files I
deal with.
Records Size
170,388 65.3MB
1,881,905 208MB
6,321,618 Comes in 3 files, 508MB, 148MB ,& 350MB (I add them
together in Access)
As you can see I'm dealing with a lot of info and its not anything I
can scale down. Its all important data. This info is refreshed about
once per month.
My Question is, what is the most efficient Database Management Program
to use for managing tons of data and doing both day to day ad hoc
reporting/excel exports (for file requests) and future organized
reporting and data mining. I have used VB a few times and am no pro at
it. I used to use Foxpro quite a bit a while back but am used to the
feel of Access 2000 mostly. Learning SQL wouldn't kill me. I am just
curious what is out there that can handle this stuff. I am already
reaching that 2GB limit on some of my Access Databases so I'm pretty
sure no matter what, Access 2000 will NOT be my back end, even if I
continue to use it as my front end.
Any help and/or comments welcome.
Thanks,
James
Christopher Browne - 23 Aug 2003 03:34 GMT
In an attempt to throw the authorities off his trail, jnovara@chasemerchantservices.com (James Novara) transmitted:
> I am currently running a 2.4ghz system with dual 40 Gig hard drives
> and 1 GB of Ram. I do an enormous amount of ad hoc reporting and once
[quoted text clipped - 6 lines]
> run my queries and the like from there. (Linking to the text files is
> near impossible.....takes forever to access the data.)
If it is being refreshed once a month, then while it may take a while
to do updates, you don't need to take that "hit" very often.
I would think that just about any decent DBMS engine should be
satisfactory. Sybase SQL Server, Firebird, PostgreSQL, Oracle, DB/2 and
such should all be fairly satisfactory.
In all cases, the more RAM you have, the better to do more caching,
and there is presumably some DBMS-tuning learning curve to go through.
They all offer reasonably powerful query engines, B-trees for pretty
quick data storage, and have some form of query optimizer (mostly
cost-based, which is preferable for large databases) so they should
cope with the data reasonably efficiently.

Signature
let name="cbbrowne" and tld="cbbrowne.com" in name ^ "@" ^ tld;;
http://cbbrowne.com/info/nonrdbms.html
"Feel free to contribute build files. Or work on your motivational
skills, and maybe someone somewhere will write them for you..."
-- "Fredrik Lundh" <effbot@telia.com>