Revived: Oninit from cron leaves engine in permanent fast

Thread view:

Enable EMail Alerts

Start New Thread

Thread rating:

Jonathan Leffler - 23 Nov 2005 21:23 GMT

Reviving some ancient history...

On 21 Oct 2005 02:54:25 -0700, gerry.cassidy@dsl.pipex.com <
gerry.cassidy@dsl.pipex.com> wrote:

> Thanks TBP, I think you may have hit on something there. The
> difference between ps outputs taken from the cron startup and the
> manual startup is shown below;
>
> < informix 585744 770152 0 02:54:26 - 0:00 oninit -v
> < informix 643312 585744 76 02:54:27 - 84:26 oninit -v
> < informix 680146 803060 0 02:54:30 - 0:00 oninit -v
> [...]
> < informix 811086 803060 0 02:54:33 - 0:00 oninit -v
> < informix 827566 803060 0 02:54:28 - 0:00 oninit -v
> ---
> > informix 643318 1 0 04:20:49 - 0:03 oninit -v
> > informix 680148 803066 0 04:20:55 - 0:00 oninit -v
> > informix 684192 803066 0 04:20:51 - 0:00 oninit -v
> > [...]
> > informix 790634 803066 0 04:20:50 - 0:00 oninit -v
> > informix 803066 643318 0 04:20:50 - 0:00 oninit -v
> > informix 827568 803066 0 04:20:53 - 0:00 oninit -v
> > informix 831622 803066 0 04:20:56 - 0:00 oninit -v
>
> I agree that the main oninit should have a PID of 1 [...]

I've finally found a bit of time to do some experimenting - using IDS
10.00.UC3 on Solaris 8.

I've entered 2 bugs and a feature request:
B175152: IDS does not properly daemonize itself
B175153: "onstat -r 60 -" should include a date/time in its output (feature
request)
B175154: "onstat -r 60 -" should terminate once it detects IDS has
terminated

The first bug shows up if you restart IDS in a shell script such as:

(
. 10.00.UC3 # Set IDS environment
onmode -ky
oninit
) | mailx -s "some subject" whoever@wherever.com

The mailx command doesn't terminate, because there are child processes of
the oninit process that still have the pipe open for writing. IDS should
detach itself from its standard input, standard output and standard error.
Consideration of 'oninit -v' indicates this should happen after the verbose
mode is complete.

The workaround is to run oninit with i/o redirection:

oninit </dev/null >/dev/null 2>&1

This permits the script to stop the server, and restart it - downside, you
don't see any diagnostics from oninit unless you poke around the online log
file.

With the workaround in place, I can bounce my server up and down via cron
quite happily - I've been doing it every ten minutes for a hour or two now,
No getting stuck in fast recovery. That means I've not been able to
reproduce your problem.

The other bug and the feature request are self-explanatory and semi-trivial
- minor usability nitpicks rather than anything outrageously awful.

Does this help anyone? Dunno; the fact that the parent process is not
terminating and leaving the system daemonized is weird. I'm wondering if
there's a setting - possibly in ONCONFIG, or possibly in the environment, or
possibly in the command line options (undocumented ones) - that tells IDS
not to daemonize itself. This sort of thing tends to be useful in R&D
debugging, and I wonder if something like a bit was set in CCFLAGS in
ONCONFIG under direction from Tech Support and was not unset afterwards, and
this is causing the misbehaviour. Exactly how are you restarting your
server?

--
Jonathan Leffler #include <disclaimer.h>
Email: jleffler@earthlink.net, jleffler@us.ibm.com
Guardian of DBD::Informix v2005.02 -- http://dbi.perl.org/
sending to informix-list

Reply to this Message

Mladen Jovanovski - 24 Nov 2005 09:14 GMT

> Reviving some ancient history...
>
[quoted text clipped - 68 lines]
> possibly in the command line options (undocumented ones) - that tells IDS
> not to daemonize itself. This sort of thing tends to be useful in R&D

I think that "oninit -d" will not daemonize oninit processes

> debugging, and I wonder if something like a bit was set in CCFLAGS in
> ONCONFIG under direction from Tech Support and was not unset afterwards, and
[quoted text clipped - 6 lines]
> Guardian of DBD::Informix v2005.02 -- http://dbi.perl.org/
> sending to informix-list

Best regards,

Signature

Mladen Jovanovski

Phone: +389 2 244 1140
Mobile: +389 75 400 309

COSMOFON - Mobile Telecommunications Services - A.D. Skopje
_______________________________________________________________
This e-mail (including any attachments) is confidential and may be protected by legal privilege. If you are not the intended recipient, you should not copy it, re-transmit it, use it or disclose its contents, but should return it to the sender immediately and delete your copy from your system. Any unauthorized use or dissemination of this message in whole or in part is strictly prohibited. Please note that e-mails are susceptible to change. COSMOFON A.D. Skopje shall not be liable for the improper or incomplete transmission of the information contained in this communication nor for any delay in its receipt or damage to your system.
sending to informix-list

Reply to this Message

gerry.cassidy@dsl.pipex.com - 29 Nov 2005 15:14 GMT

Hi Jonathan

Thanks for the reply and bug reports.

I have tried your workaround but to no avail.

Our problem is that the restart of IDS does work correctly within a
script, even without the redirection (using 9.40.FC6 so your bug may be
specific to v10.UC3).

I have also tried the suggestion by Mladen to use the (undocumented)
method of starting the engine with oninit -d. This does not daemonize
the engine processes but runs in the foreground. However the databases
within the server are usable.

When I use oninit -d in the script, called from cron, the result is the
same ... Fast Recovery that does not recover!

Thanks again to both of you but no joy :(

Gerry

> Reviving some ancient history...
>
[quoted text clipped - 78 lines]
> Guardian of DBD::Informix v2005.02 -- http://dbi.perl.org/
> sending to informix-list

Reply to this Message

david@smooth1.co.uk - 29 Nov 2005 18:42 GMT

Run oninit with an & after it.

oninit > myfile 2>&1 &

That will background it for sure!

Reply to this Message

Jonathan Leffler - 30 Nov 2005 07:44 GMT

> Run oninit with an & after it.
>
> oninit > myfile 2>&1 &
>
> That will background it for sure!

You omitted the '-d', which is part of the answer. There are
occasions when R&D or Tech Support would find the '-d' option useful
but very few occasions when a regular customer would use it -
especially not in production.

You should also redirect standard input - probably to /dev/null:

(oninit </dev/null >/dev/null 2>&1) &

I use the parentheses to run a sub-shell in the background, and that
is responsible for running oninit. If you want to, you can include a
second & on its own inside the parentheses after the 1, but it is
hardly necessary.

The i/o redirection there means that none of the standard I/O streams
(stdin, stdout, stderr) is connected to the terminal or pipe, so you
should be OK unless you have some weird piece of code somewhere that
plays silly buggers with i/o redirection - like doing:

exec 1>&3

If that came before the command above, that would keep the original
stdout open despite the i/o redirection I showed.

If this doesn't solve the problem, then we may have to resort to some
tricky stuff. If you have the lsof (list of open files) program,
maybe you should run that in the cron-run script to a file and
establish which files are open. Failing that, I have a poor man's
version of 'lsof'; the current incarnation just lists output like
'ooo---o----o--' to show that file descriptors 0, 1, 2, 6, and 11 (if
I counted correctly) are open and the others are closed. You'd expect
that to produce 'ooo-----------' for a cron-run job; if there are any
other files open, they could be a source of trouble. A fairly easy
enhancement to the program would be to print the device information
(file inode number and device major/minor) for each open file; that
approaches 'lsof' more closely.

Gerry Cassidy wrote:

>When I use oninit -d in the script, called from cron, the result is the
>same ... Fast Recovery that does not recover!

Have you really, really tracked all the environment - including things
like current directory and umask and whatnot - in the script run by
cron? Have you done a line-by-line comparison of the environment for
the cron-run script with your command-line environment? Have you
tried using 'env -i' and setting explicitly the Informix environment
variables that are truly necessary in your machine? I set just 7 of
them: INFORMIXDIR, INFORMIXSERVER, ONCONFIG, HOME, PATH, SHELL, TZ.
IDS runs nicely for me - and did when run from a cron script too.

--
Jonathan Leffler #include <disclaimer.h>
Email: jleffler@earthlink.net, jleffler@us.ibm.com
Guardian of DBD::Informix v2005.02 -- http://dbi.perl.org/
sending to informix-list

Reply to this Message

david@smooth1.co.uk - 30 Nov 2005 22:43 GMT

Well if there is something then shouldn't IDS put a message in the
online.log to say it
has spotted this flag?

http://www.prstech.com/src/informix_faq/ifaq08.htm

"Environment variables to set before starting Online:-

AIOTRACE=1 Writes additional message to online.log"

and

"Another one is if you are using DR and want to see what's going on
between servers set DR_TRACE=1
and pipe oninit to a file."

And what do the CCFLAGS bits mean? Any plans to tell us?

Reply to this Message