Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
Home
Discussion Groups
Database Servers
DB2InformixIngresMS SQLOraclePervasive.SQLPostgreSQLProgressSybase
Desktop Databases
FileMakerFoxProMS AccessParadox
General
General DB TopicsDatabase Theory
Related Topics
Java Development.NET DevelopmentVB DevelopmentMore Topics ...

Database Forum / Informix Topics / December 2004

Tip: Looking for answers? Try searching our database.

DB Load Performance

Thread view: 
Enable EMail Alerts  Start New Thread
Thread rating: 
bothwellno@spam.duke.edu - 30 Dec 2004 19:04 GMT
I'm helping a group with a database reorganization.
System info:
4-way 332 MHz IBM F50, 3GB RAM
Local SCSI disks for OS & file system
2 RAID-10 arrays on separate SSA enclosures
AIX 5.1 running IDS 9.30.UC1

This group used constraints to create primary keys (53) and foreign
keys (65).
I've taken the constraints out of the schema and put them in a separate
query to run after the data is loaded.
I've added indexes for all of the constraints to allow me to put them
in dbspaces separate from the data.

I use HPL to load the tables (55) in about 1 hour.
The indexes (125) take just under 3 hours to load.

The sql script to create the constraints (118) takes 17 hours to run.
After the constraints are done, I use dostats to run update
statistics - it takes about 8 hours to run.

I'd like to optimize these last 2 items, especially the constraints
part.
Here is some information about the database:

Data: 8.5 GB
Indexes: 13 GB

One table (xlarge) occupies the most space (data: 6 GB, indexes: 8GB)

xlarge table details:
Rows: 142,128,896

{ TABLE "informix".xlarge row size = 61 number of columns = 13 index
size = 81 }
create table "informix".xlarge
 (
   process_id integer not null ,
   claim_seq smallint not null ,
   payer_id char(6),
   route_id smallint,
   service_id char(2),
   payer_group char(2),
   pat_ctrl_num varchar(30),
   pay_seq char(1),
   paper_status char(1),
   proc_status char(1),
   special_type char(1),
   batch_id integer,
   claim_tot_amt integer,
   primary key (process_id,claim_seq)  constraint "informix".pk_xlarge
 ) in xlarge_dbs extent size 10315804 next size 1031580 lock mode page;

create unique index "informix".u1_xlarge on "informix".xlarge
(process_id,claim_seq)
 using btree in xlarge_idx;
create index "informix".ic1_xlarge on "informix".xlarge (payer_id)
 using btree in xlarge_idx;
create index "informix".ic2_xlarge on "informix".xlarge (process_id)
 using btree in xlarge_idx;
create index "informix".ic3_xlarge on "informix".xlarge
(batch_id,route_id,service_id)
 using btree in xlarge_idx;
create index "informix".ic4_xlarge on "informix".xlarge
(route_id,service_id)
 using btree in xlarge_idx;
create index "informix".ic5_xlarge on "informix".xlarge (payer_group)
 using btree in xlarge_idx;

alter table xlarge add constraint primary key (process_id,claim_seq)
 constraint pk_xlarge;
alter table "informix".xlarge add constraint (foreign key (payer_id)
 references "informix".payer constraint "informix".fk_referen_ident_2);
alter table "informix".xlarge add constraint (foreign key (process_id)
 references "informix".t_process on delete cascade
 constraint "informix".fk_ref_pro_track_p);
alter table "informix".xlarge add constraint (foreign key
(batch_id,route_id,service_id)
 references "informix".t_batch constraint
"informix".fk_referen_track_b);
alter table "informix".xlarge add constraint (foreign key
(route_id,service_id)
 references "informix".route constraint "informix".fk_ref_rou_route2);
alter table "informix".xlarge add constraint (foreign key (payer_group)
 references "informix".p_group constraint
"informix".fk_ref_pay_payer_g);

Here are some constraints on other tables that reference the xlarge
table:

alter table "informix".c_report add constraint (foreign key
(process_id,claim_seq)
 references "informix".xlarge on delete cascade constraint
"informix".fk_ref_t_c_track_c);
alter table "informix".tc_proc add constraint (foreign key
(process_id,claim_seq)
 references "informix".xlarge on delete cascade constraint
"informix".fk_ref_tra_track_c);

Each of these foreign key constraints takes just over 2 hours to run
(about 14 of the 17 total hours for all of the alters).
I've played around with BUFFERS and SHMVIRTSIZE without any luck.
Any ideas on how to speed up the creation of these constraints?

It took 5.5 hours to run update statistics on this table.
I run 2 'update statistics' scripts for the whole database, one for the
xlarge table and another for the rest of the tables.
Can the 'update statistics' for the xlarge table be broken down into
separate scripts for each index column and run in parallel, or will
I run into locking issues?

Thanks.

Bob

Signature

bothwellno@spam.duke.edu

Art S. Kagel - 30 Dec 2004 21:35 GMT
Check out John F. Miller III's paper, 'Understanding and Tuning Update
Statistics' much in there is also applicable to index builds, it's online in
the IBM library, and I believe that John is giving that talk at the
IIUG/IDUG conference in May and at the next WAIUG meeting.  In addition,
note that once set up you rarely have to run a full dostats run.  If you
have a recent version (or download it) check out the -a/A and -b/B options
which will schedule subsets of your tables for update only when needed.
Over time you will find running dostats with these options set in a cron
overnight or over the weekend will tend to update only a few tables per run.

In general, turn on PSORT_NPROCS set to 2xNUMCPUVPS, PSORT_DBTEMP set to a
list of 3-6 large fast filesystems (light FSes are perfect!) with enough
free space each to hold all sort-work data for a table.  PDQPRIORITY
maximized to as close to 100% as you dare without starving users in other
databases.  Set these during index builds, dostats runs (also configure
DBUPSPACE per John's paper), and especially constraint building/enablement.

Art S. Kagel

> I'm helping a group with a database reorganization.
> System info:
[quoted text clipped - 110 lines]
>
> Bob
 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2009 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.