Backup and OS Swapping

Thread view:

Enable EMail Alerts

Start New Thread

Thread rating:

mike_dba - 26 Jun 2006 21:27 GMT

I have recently upgraded from SuSe 8 to 9 (kernel 2.4 to kernel 2.6).
I am running DB2 8.1 FP 11 (aka 8.2 FP 4). I have had slowdowns that I
suspect are related to the fact that an online backup is being run.
Can someone help me understand the behaviour described below and offer
a suggestion as to how to tune this ? Your assistance is appreciated.

I have a system running at 95 idle before the backup starts. Once the
backup is underway, I see a split in vmsat of roughtly 45% system and
50% wait. I also see low run-q but free memory goes down significantly
and swapping in and out is occurring. Since this is an online backup,
I do have queries coming in but do not suspect that they are causing
this as it is off-hours and the snapshots do not indicate anything out
of the ordinary with them.

I have plenty of memory on the box - some 16 GB with 3 GB for
bufferpool and 3 GB for sort allocated.

Thanks.

Reply to this Message

Ian - 27 Jun 2006 13:55 GMT

> I have recently upgraded from SuSe 8 to 9 (kernel 2.4 to kernel 2.6).
> I am running DB2 8.1 FP 11 (aka 8.2 FP 4). I have had slowdowns that I
[quoted text clipped - 14 lines]
>
> Thanks.

Where are you sending the backup (i.e. local disk, NFS file system,
etc)?

Can you post the vmstat output while this is occurring?

Reply to this Message

mike_dba - 27 Jun 2006 14:19 GMT

> > I have recently upgraded from SuSe 8 to 9 (kernel 2.4 to kernel 2.6).
> > I am running DB2 8.1 FP 11 (aka 8.2 FP 4). I have had slowdowns that I
[quoted text clipped - 19 lines]
>
> Can you post the vmstat output while this is occurring?

Ian,

Thanks for the reply. This is posing a big problem as simple queries
degrade significantly while the is occurring.

The backup is going to local disk. I am am running EEE (DPF). The
script rsh the backup command to each host. It backs up each partition
to a disk local to each host.

The linux 'uptime' and vmstat on one of the hosts looks like this at
the start -

cat vmstat.njds0052.11:00

11:00am up 19 days 8:42, 2 users, load average: 0.79, 1.31, 1.17
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----
r b swpd free buff cache si so bi bo in cs us
sy id wa
0 0 445736 898452 565076 13486952 7 8 12482 753 12 9 23
4 64 9
0 2 445736 1738696 565736 12646416 0 0 648 98 1612 8913
12 36 38 14
0 2 445736 1736836 567604 12645576 0 0 1874 64 1596 3446
5 11 44 40
0 3 445736 1735192 569156 12646080 0 0 1418 620 1898 12300
9 10 37 43
0 3 445736 1733704 570676 12645588 0 0 1512 72 1749 4205
3 8 47 42

Here we are 10 minutes into it and you can see the script that runs
every 10 min actually took 2 add'l mins (11:12)

cat vmstat.njds0052.11:12

11:12am up 19 days 8:54, 2 users, load average: 30.43, 17.71, 8.49
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----
r b swpd free buff cache si so bi bo in cs us
sy id wa
5 10 2032160 35484 97064 14231460 7 8 12487 762 12 10
23 4 64 9
1 6 2033268 36680 98272 14227160 1758 244 164476 96230 1860 6059
9 59 1 32
2 4 2033956 39184 98208 14228028 466 424 164120 77460 1745 7437
9 59 2 30
1 5 2035300 36704 98636 14227800 0 528 168542 59732 1651 3887
6 53 2 39
9 4 2037012 38068 99140 14225460 0 974 170332 57634 1671 3842
6 53 3 38

And later,
cat vmstat.njds0052.11:40

11:40am up 19 days 9:22, 3 users, load average: 23.22, 23.32,
20.56
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----
r b swpd free buff cache si so bi bo in cs us
sy id wa
1 9 3288968 35356 133008 14801316 8 9 12508 796 13 2
23 4 64 9
3 8 3287552 35480 133792 14803696 1362 386 81426 88628 1739 5788
4 37 5 54
2 13 3286592 39820 134344 14801520 1834 508 46990 95744 1828 8214
3 24 8 65
1 13 3285016 38324 135120 14798768 1768 400 88354 60472 2071 6245
6 47 4 43
2 19 3280004 41216 135164 14796956 4470 6 7018 37188 1792 5374
2 6 7 85

cat vmstat.njds0052.11:50

11:50am up 19 days 9:32, 3 users, load average: 14.25, 14.76,
17.48
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----
r b swpd free buff cache si so bi bo in cs us
sy id wa
0 3 1516436 37560 135896 15174532 8 9 12511 801 0 4
23 4 64 9
0 2 1516196 36196 136452 15176536 184 4 53784 29938 7249 7595
40 28 8 24
29 2 1516000 47284 136188 15150748 208 48 29968 72668 5789 4395
69 26 0 4
45 2 1515968 40340 136056 15154360 96 6 43782 74056 2745 897
80 20 0 0
46 3 1515900 43300 136392 15146004 82 4 44694 23618 3049 1576
80 20 0 0

cat vmstat.njds0052.12:00 <--- Here ware have completed

12:00pm up 19 days 9:42, 3 users, load average: 0.07, 3.07, 10.27
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----
r b swpd free buff cache si so bi bo in cs us
sy id wa
0 0 1432332 276984 177516 14924624 8 9 12509 803 1 4
23 4 64 9
0 2 1431708 275752 177516 14924952 662 0 1656 20 1192 1648
1 0 82 18
0 1 1429304 272032 177516 14927840 1814 0 2810 144 1538 2269
1 1 50 49
1 2 1427752 269668 177532 14929072 1180 0 1750 210 1421 2394
1 1 51 46
0 2 1426076 266568 177540 14930836 1494 0 2172 144 1370 2330
0 1 50 49

I also have the following 'ps -ef' output -

cat top20.njds0052.11:12
UID PID %CPU CPU S TIME COMMAND
1001 8926 1.8 - S 00:00:07 db2med.18636.3 2

1001 8932 1.8 - S 00:00:06 db2med.18636.9 2

1001 8918 1.6 - S 00:00:06 db2bm.18636.1 2

1001 8920 1.6 - S 00:00:06 db2bm.18636.5 2

1001 8924 1.6 - S 00:00:06 db2med.18636.1 2

1001 8923 1.6 - S 00:00:06 db2med.18636.0 2

1001 8928 1.6 - S 00:00:05 db2med.18636.5 2

1001 8931 1.5 - S 00:00:05 db2med.18636.8 2

1001 8927 1.5 - S 00:00:05 db2med.18636.4 2

1001 8929 1.4 - S 00:00:05 db2med.18636.6 2

1001 8925 1.4 - S 00:00:05 db2med.18636.2 2

1001 8930 1.4 - S 00:00:05 db2med.18636.7 2

1001 15950 1.1 - S 00:01:45 db2agent (CIF) 1

1001 32014 1.1 - S 00:27:02 db2fcmdm 1

1001 10034 1.0 - S 00:00:03 db2med.15950.7 1

1001 32009 1.0 - S 00:24:54 db2fcmdm 2

1001 10031 0.8 - S 00:00:02 db2med.15950.4 1

1001 10028 0.7 - S 00:00:02 db2med.15950.1 1

1001 10036 0.7 - S 00:00:02 db2med.15950.9 1

cat top20.njds0052.11:20
UID PID %CPU CPU S TIME COMMAND
0 12048 3.0 - S 00:00:00 [pdflush]
1001 12052 2.0 - R 00:00:00 ps -e -o
uid,pid,pcpu,cpu,state,cputime,args --sort -pcpu,-time
1001 8926 1.2 - D 00:00:11 db2med.18636.3 2

1001 8932 1.2 - D 00:00:10 db2med.18636.9 2

1001 8918 1.2 - S 00:00:10 db2bm.18636.1 2

1001 8923 1.2 - D 00:00:10 db2med.18636.0 2

1001 8928 1.2 - S 00:00:10 db2med.18636.5 2

1001 8924 1.1 - D 00:00:10 db2med.18636.1 2

1001 8920 1.1 - S 00:00:10 db2bm.18636.5 2

1001 8931 1.1 - D 00:00:10 db2med.18636.8 2

1001 8930 1.1 - D 00:00:09 db2med.18636.7 2

1001 8927 1.1 - D 00:00:09 db2med.18636.4 2

1001 15950 1.1 - S 00:01:45 db2agent (CIF) 1

1001 8929 1.1 - D 00:00:09 db2med.18636.6 2

1001 32014 1.1 - D 00:27:03 db2fcmdm 1

1001 8925 1.1 - D 00:00:09 db2med.18636.2 2

1001 32009 1.0 - S 00:24:55 db2fcmdm 2

0 10871 0.8 - S 00:00:03 [pdflush]
1001 10034 0.8 - D 00:00:06 db2med.15950.7 1

cat top20.njds0052.11:31
UID PID %CPU CPU S TIME COMMAND
0 13862 3.0 - D 00:00:04 [pdflush]
0 13912 2.5 - D 00:00:01 [pdflush]
0 13904 2.4 - S 00:00:02 [pdflush]
0 13925 1.9 - D 00:00:00 [pdflush]
0 14020 1.3 - D 00:00:00 [pdflush]
0 14023 1.2 - D 00:00:00 [pdflush]
0 13942 1.2 - D 00:00:00 [pdflush]
1001 8918 1.1 - S 00:00:17 db2bm.18636.1 2

1001 8926 1.1 - D 00:00:17 db2med.18636.3 2

1001 8932 1.1 - D 00:00:17 db2med.18636.9 2

1001 8923 1.1 - S 00:00:17 db2med.18636.0 2

1001 8928 1.1 - D 00:00:16 db2med.18636.5 2

1001 8931 1.1 - D 00:00:16 db2med.18636.8 2

1001 32014 1.1 - D 00:27:04 db2fcmdm 1

1001 8930 1.1 - D 00:00:16 db2med.18636.7 2

1001 8927 1.1 - D 00:00:16 db2med.18636.4 2

1001 8920 1.1 - S 00:00:16 db2bm.18636.5 2

1001 8924 1.1 - D 00:00:16 db2med.18636.1 2

1001 8929 1.0 - D 00:00:16 db2med.18636.6 2

Reply to this Message