EDU process dieing

Thread view:

Enable EMail Alerts

Start New Thread

Thread rating:

soccertl - 21 Nov 2006 17:49 GMT

We have a program running on AIX and uses DB/2. Every once in a while,
while our program is just running along, DB/2 dies due to a signal 9
being sent to one of the EDU processes:

2006-11-14-12.26.23.442685+000 E396314C794 LEVEL: Severe
PID : 72190 TID : 1 PROC : db2ipccm 0
INSTANCE: hdbuser NODE : 000
FUNCTION: DB2 UDB, oper system services, sqloEDUSIGCHLDHandler,
probe:50
DATA #1 : <preformatted>
Detected the death of an EDU with process id 141222
The signal number that terminated this process was 9
Look for trap files (t141222.*) in the dump directory
CALLSTCK:
[0] 0xD4141828 sqloEDUSIGCHLDHandler__FiP9siginfo_tPv + 0x220
[1] 0x00000000 ?unknown + 0x0
[2] 0x00000000 ?unknown + 0x0
[3] 0x00000000 ?unknown + 0x0
[4] 0x00000000 ?unknown + 0x0
[5] 0x00000000 ?unknown + 0x0
[6] 0x00000000 ?unknown + 0x0
[7] 0x00000000 ?unknown + 0x0
[8] 0x00000000 ?unknown + 0x0
[9] 0x00000000 ?unknown + 0x0

DB/2 support is no help on this as they just say someone is sending
them the signal. We have no code to send this signal to a DB/2 process
and nobody is killing anything from the command line. Our process is
just running along accessing the database. So, we are stuck unless we
can somehow trace who might be doing this.

Is there something we can set up in DB/2 or AIX that would help us
determine who is sending this signal?

Thanks for any help or suggestions.

Reply to this Message

soccertl - 21 Nov 2006 18:08 GMT

Also, levels are:

AIX 5.3.0.50
DB/2 8.1.1.120 FixPak 13

Reply to this Message

Knut Stolze - 21 Nov 2006 20:21 GMT

> We have a program running on AIX and uses DB/2. Every once in a while,
> while our program is just running along, DB/2 dies due to a signal 9
[quoted text clipped - 9 lines]
> The signal number that terminated this process was 9
> Look for trap files (t141222.*) in the dump directory

Have you such trap files in the dump directory? If so, what's the content?

> DB/2 support is no help on this as they just say someone is sending
> them the signal.

Right. DB2 (not DB/2!) doesn't send the SIGKILL signal to its processes.
So someone else must be doing this and you should figure out what's going
on in your environment.

Signature

Knut Stolze
DB2 Information Integration Development
IBM Germany

Reply to this Message

soccertl - 21 Nov 2006 20:38 GMT

> Have you such trap files in the dump directory? If so, what's the content?

No. Those files do not exist. DB2 support said they would not be there
for some reason I can't remember.

> Right. DB2 (not DB/2!) doesn't send the SIGKILL signal to its processes.
> So someone else must be doing this and you should figure out what's going
> on in your environment.

Right. The problem is trying to figure out who that "someone else" is
since our code doesn't send SIGKILL to DB2 processes either. I was just
wondering if someone had any insight into what I could use to track
this down.

Reply to this Message

Larry - 21 Nov 2006 21:27 GMT

I would ask your UNIX admins whether they have a way to trace who issues
"kill -9" commands.

Larry Edelstein

>>Have you such trap files in the dump directory? If so, what's the content?
>
[quoted text clipped - 9 lines]
> wondering if someone had any insight into what I could use to track
> this down.

Reply to this Message

Phil Sherman - 23 Nov 2006 03:39 GMT

A quick & dirty way to do this is to replace the kill command with a
script that logs the date, time, and user then issues the kill command.
If the user doing this is using sudo to execute the kill command, then
you'll need to check the appropriate sudo logs.

Your UNIX administrator should be able to do this in less than 10 min.

Phil Sherman

> I would ask your UNIX admins whether they have a way to trace who issues
> "kill -9" commands.
[quoted text clipped - 17 lines]
>> wondering if someone had any insight into what I could use to track
>> this down.

Reply to this Message

soccertl - 28 Nov 2006 19:52 GMT

> A quick & dirty way to do this is to replace the kill command with a
> script that logs the date, time, and user then issues the kill command.
> If the user doing this is using sudo to execute the kill command, then
> you'll need to check the appropriate sudo logs.

Thanks for the idea. I will check with our AIX install developers.

> Your UNIX administrator should be able to do this in less than 10 min.

There is no UNIX administrator. This is a development lab with
development machines. We have developers in charge of the AIX install
but that is as close as we come. I can ask them to look into doing your
suggestion.

Reply to this Message

soccertl - 21 Nov 2006 20:43 GMT

> Right. DB2 (not DB/2!)

And sorry. I was used to the OS/2 world where it was DB2/2 and I have a
hard time removing the '/'. :)

Reply to this Message