Hi Marty,

That was exactly the sickly warm feeling I was getting yesterday about
this whole episode.  Might raise a DAR to get the rollforwarddb command
to check the database name passed to it against that contained in the
config file and put out a confirmation message if they don't match.
Alternatively, I'll finish writing my rollforwarddb wrapper to handle
all the copying of config files, etc.  I have to say that I ran infodb
after copying the config file but didn't pay close enough attention to
the output - that comes from doing a 36 hour shift I guess.

Not sure how successful the force consistent and -e flag would have
been.  I think the first journal file had been corrupted but I could be
wrong there.

Jon

PS No random duckman quote??
-----Original Message-----
From: martin.bowes@ctsu.ox.ac.uk [mailto:martin.bowes@ctsu.ox.ac.uk]
Sent: 18 January 2005 11:04
To: Gibson Jonathan
Cc: info-ingres@cariboulake.com
Subject: Re: [Info-ingres] Rollforwarddb disaster

Hi Jono,

    I'm not 100% sure about this, but I suspect that...

    The DMP Config file controls the recovery once it gets going.

    So you had in the DMP(imadb) the config file for another
database. The system has stalled imadb correctly.  But now uses the
details contained in the DMP config file to control the rest. This is
required as the DATA config file is about to be obliterated by the
recovery itself. Hence the rename of the data config to aaaaaaaa.rfc.

    I'm not surprised that the recovery didn't at least check that
the
two files were similar. The recovery is meant to handle some fairly
nasty conditions so I suppose they've gone for a minimalist approach.

    As a consequence the LIVE database got severely beaten up.
It was not correctly stalled so God only knows what could have
happened to it - or to the Journals being written whilst all of this was

going on. Hence I suspect that the system also would have realised
that the journal files in the production system could now have contained

all sorts of gibberish and would have 'marked' them as suspect.

    I suspect (conjecture?) that your only way to have recovered
reasonably fully from here would have been to use verifydb to force the
database logically consistent, then rollforwarddb with a -e flag.

    Marty

want to mention (I feel a

aaaaaaaa.cnf in the dmp location of

recovery process. When the

database for preparation

the system. After a few

finished and we then noticed

Needless to say, the users

replaced in front of

and restore both the LIVE

no matter what we tried,

since been recovered to

(the business was happy to

easily, rather than struggle for

LIVE database, even with a

in some way (it never got

disk activity for over an hour but

recovery of database that

this from happening.

and part of this problem

running again.

--
Random Farscape Quote #27:
Rygel - I don't believe the Universe will follow my preconceptions, but
I
       know a fact when it hits me in the face.

_____________________________________________________________________
This message has been checked for all known viruses by bluesource. For
further information visit www.blue-source.com 

powered by Messagelabs

<font size='1' face='arial'>**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. No one else is authorised to distribute, forward,
print, copy or act upon any information contained in this email.
If you have received this email in error, please notify the sender.

Hiscox Syndicates Limited, Hiscox Insurance Company Limited,
Hiscox Connect Limited, Hiscox Underwriting Limited and
Hiscox Investment Management Limited are authorised and
regulated by the Financial Services Authority. Hiscox plc is a
company registered in England and Wales under company
registration number 2837811 and registered office at
1 Great St Helen's, London EC3A 6HX
**********************************************************************

_____________________________________________________________________
This message has been checked for all known viruses by blue-source. For further information visit www.blue-source.com

powered by Messagelabs

I did a similar thing trying to restore a production checkpoint to test.
I copied the ckp and config to test then ran the restore. It wiped out
production whilst the users were on line. I was lucky. A rollforwarddb -j -e
fixed it.
Paul

-----Original Message-----
From: martin.bowes@ctsu.ox.ac.uk [mailto:martin.bowes@ctsu.ox.ac.uk]
Sent: Tuesday, 18 January 2005 10:33 PM
To: Gibson Jonathan; info-ingres@cariboulake.com
Subject: RE: [Info-ingres] Rollforwarddb disaster

Hi Jon,

    36hours...Oh that explains a lot!

    I thought the Farscape quote was appropriate myself. But just
in case you prefer the wisdom of duckman.

    Marty
PS. I'm hoping to come across some pirated Duckman DVDs very
shortly. I'm sure the Duck would approve.

--
Random Duckman Quote #46:
Duckman: Damn I must have missed one...Um, I mean cram, unjust,
math, piston.
Cornfed: Uh huh. Are you having a stroke?
Duckman: I haven't decided yet.

*******

_______________________________________________
Info-ingres mailing list
Info-ingres@cariboulake.com
http://mailman.cariboulake.com/mailman/listinfo.py/info-ingres

I really think there should be a check in the rollforwarddb procedure at
the point it starts to read the config file.  This should verify that
the dbname supplied to the rollforwarddb command matches that found in
the config file.  If it doesn't, then it should warn the user before
commencing to allow the whole procedure to be cancelled in case it is a
genuine mistake.  I've been impressed with Ingres over the years but
this to me is a major flaw in the code and was pretty close to costing
me my job.  A DAR shall be winging it's way to CA today.

-----Original Message-----
From: Paul White [mailto:pwhite@peerlessit.com.au]
Sent: 18 January 2005 23:43
To: 'martin.bowes@ctsu.ox.ac.uk'; Gibson Jonathan;
info-ingres@cariboulake.com
Subject: RE: [Info-ingres] Rollforwarddb disaster

I did a similar thing trying to restore a production checkpoint to test.
I copied the ckp and config to test then ran the restore. It wiped out
production whilst the users were on line. I was lucky. A rollforwarddb
-j -e
fixed it.
Paul

-----Original Message-----
From: martin.bowes@ctsu.ox.ac.uk [mailto:martin.bowes@ctsu.ox.ac.uk]
Sent: Tuesday, 18 January 2005 10:33 PM
To: Gibson Jonathan; info-ingres@cariboulake.com
Subject: RE: [Info-ingres] Rollforwarddb disaster

Hi Jon,

    36hours...Oh that explains a lot!

    I thought the Farscape quote was appropriate myself. But just
in case you prefer the wisdom of duckman.

    Marty
PS. I'm hoping to come across some pirated Duckman DVDs very
shortly. I'm sure the Duck would approve.

--
Random Duckman Quote #46:
Duckman: Damn I must have missed one...Um, I mean cram, unjust,
math, piston.
Cornfed: Uh huh. Are you having a stroke?
Duckman: I haven't decided yet.

command

be

was

contained

the

don't

in

process.

and

out

has

replayed

the

journals

prevents

old

upgrade.

but

****
*******

further information visit www.blue-source.com

_______________________________________________
Info-ingres mailing list
Info-ingres@cariboulake.com
http://mailman.cariboulake.com/mailman/listinfo.py/info-ingres

_____________________________________________________________________
This message has been checked for all known viruses by bluesource. For
further information visit www.blue-source.com 

powered by Messagelabs

_____________________________________________________________________
This message has been checked for all known viruses by blue-source. For further information visit www.blue-source.com

powered by Messagelabs

Hi everyone,

Well, I raised a support call with CA and they claim this isn't a bug as
the copying of the config file is an unsupported feature.  I guess
they're right on this but they did suggest to raise a suggestion for the
database in the config file to be checked against that provided with the
recovery command.  In fact, they suggested to apply this to any action
that reads the config file. Fair enough, I shall go ahead and do this
but can I ask what other sites generally do to recover a database.  I'm
wondering if I'm confusing a disaster recovery scenario, such as a lost
disk where the copying of the config file may be required, as opposed to
a routine recovery of the database.

Cheers
Jon

-----Original Message-----
From: martin.bowes@ctsu.ox.ac.uk [mailto:martin.bowes@ctsu.ox.ac.uk]
Sent: 19 January 2005 10:30
To: Gibson Jonathan
Cc: info-ingres@cariboulake.com
Subject: RE: [Info-ingres] Rollforwarddb disaster

Agreed its a bug of omission and a bad one. But...

Dont raise it as a DAR, raise it as a Bug with severity level 2.

I'll bet that you'll get a patch solution faster that way!

Marty
--
Random Farscape Quote #19:
John - We don't understand the R2D2 Crap, lets use the Startrek system.
One
      blink for 'yes', two for 'no'.

_____________________________________________________________________
This message has been checked for all known viruses by bluesource. For
further information visit www.blue-source.com 

powered by Messagelabs

<font size='1' face='arial'>**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. No one else is authorised to distribute, forward,
print, copy or act upon any information contained in this email.
If you have received this email in error, please notify the sender.

Hiscox Syndicates Limited, Hiscox Insurance Company Limited,
Hiscox Connect Limited, Hiscox Underwriting Limited and
Hiscox Investment Management Limited are authorised and
regulated by the Financial Services Authority. Hiscox plc is a
company registered in England and Wales under company
registration number 2837811 and registered office at
1 Great St Helen's, London EC3A 6HX
**********************************************************************

_____________________________________________________________________
This message has been checked for all known viruses by blue-source. For further information visit www.blue-source.com

powered by Messagelabs

Hi all,

From reading this thread, it sounds to me like the database that was
originally being recovered was within the same Ingres installation as
the Production database.  This then lead to the possibility of the
config file incorrectly referencing the wrong (production) database
during the recovery scenario.  If it's feasible at your site, I'd
recommend having 2 installations of Ingres (ideally on 2 separate
machines) to avoid that possibility.  At my site we have our Production
machine and installation of Ingres with the one production database
within it.  We have a separate Development machine with its own
installation of Ingres and various development databases within that.
With this scheme, when I've had to perform disaster recovery in
Production before, there was never any possibility of accessing the
wrong database with the config file.

I also refresh my development environment from the production
checkpoint, and during that process I bring over the production config
file.  Since it's a separate Ingres installation, there isn't any
possibility that I'll screw up and affect Production during the
rollforward phase of the Development refresh process.

Hope this helps,
Troy

Troy E. Anderson
BV SOLUTIONS GROUP, Inc., A Black & Veatch Company
10950 Grandview Ave.
Overland Park, KS 66210
Tel. (913) 458-8521
Fax. (913) 458-4994
<mailto: andersonte@bvsg.com>
Visit us: http://www.bvsg.com 



-----Original Message-----
From: info-ingres-admin@cariboulake.com
[mailto:info-ingres-admin@cariboulake.com] On Behalf Of Gibson Jonathan
Sent: Wednesday, January 19, 2005 8:42 AM
To: martin.bowes@ctsu.ox.ac.uk
Cc: info-ingres@cariboulake.com
Subject: RE: [Info-ingres] Rollforwarddb disaster

Hi everyone,

Well, I raised a support call with CA and they claim this isn't a bug as
the copying of the config file is an unsupported feature.  I guess
they're right on this but they did suggest to raise a suggestion for the
database in the config file to be checked against that provided with the
recovery command.  In fact, they suggested to apply this to any action
that reads the config file. Fair enough, I shall go ahead and do this
but can I ask what other sites generally do to recover a database.  I'm
wondering if I'm confusing a disaster recovery scenario, such as a lost
disk where the copying of the config file may be required, as opposed to
a routine recovery of the database.

Cheers
Jon

-----Original Message-----
From: martin.bowes@ctsu.ox.ac.uk [mailto:martin.bowes@ctsu.ox.ac.uk]
Sent: 19 January 2005 10:30
To: Gibson Jonathan
Cc: info-ingres@cariboulake.com
Subject: RE: [Info-ingres] Rollforwarddb disaster

Agreed its a bug of omission and a bad one. But...

Dont raise it as a DAR, raise it as a Bug with severity level 2.

I'll bet that you'll get a patch solution faster that way!

Marty
--
Random Farscape Quote #19:
John - We don't understand the R2D2 Crap, lets use the Startrek system.
One
      blink for 'yes', two for 'no'.

_____________________________________________________________________
This message has been checked for all known viruses by bluesource. For
further information visit www.blue-source.com 

powered by Messagelabs

<font size='1'
face='arial'>***********************************************************
***********
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. No one else is authorised to distribute, forward,
print, copy or act upon any information contained in this email.
If you have received this email in error, please notify the sender.

Hiscox Syndicates Limited, Hiscox Insurance Company Limited,
Hiscox Connect Limited, Hiscox Underwriting Limited and
Hiscox Investment Management Limited are authorised and
regulated by the Financial Services Authority. Hiscox plc is a
company registered in England and Wales under company
registration number 2837811 and registered office at
1 Great St Helen's, London EC3A 6HX
**********************************************************************

_____________________________________________________________________
This message has been checked for all known viruses by blue-source. For
further information visit www.blue-source.com

powered by Messagelabs

_______________________________________________
Info-ingres mailing list
Info-ingres@cariboulake.com
http://mailman.cariboulake.com/mailman/listinfo.py/info-ingres

Marty writes...

have yet to see one come to fruition!

I have seen a half dozen of my DARs (OpenROAD and Ingres) DARs come to
fruition in the past 5 years. One if the ingres DARs came out in a patch in
less than a week.  Well, now they're called suggestions. Yes, we have to
lobby a bit harder now, but you've got to log them otherwise they go to the
bottom of the list. Try it.

Paul

ps. I would like to easily restore a checkpoint to a different database.
What about a parameter to specify the source and destination databases? Then
we could be sure the restore is not going to clobber the production DB.
Somthing like:

rollforwarddb -source production -dest test

This would be really helpful for those requests "can you please restore a
copy of my table from last week - but don't overwrite production ?"

Hi Paul,

I don't understand why one utility, ie infodb, looks at the config file
in the data location and another utility, rollforwarddb looks in the
config file in the dmp location.  I'm sure there's a very good reason
for it.  Anyway, at the request of the my IT director, I've built a
wrapper that performs a "strings aaaaaaaa.cnf" in the dmp location to
get the dbname from the config file and makes sure that it reconciles
with the dbname passed with the rollforwarddb command and bombs out if
it doesn't match.  Can't say I'd ever make the mistake again but it's
nice to have a little security blanket to keep me in a job.

Yup, you're right that I did copy the dmp config to the data config.  If
this is wrong, even for a full disaster recovery scenario, then I'll
amend our procedures appropriately.  I think I must have been hungover
at the IUA conference in Blackpool when Karl gave his talk on recovery.
I picked up quite a few good tips from that presentation that helped us
do a successful DR test but I must have got a crossed wire somewhere ;-)

Jon

-----Original Message-----
From: info-ingres-admin@cariboulake.com
[mailto:info-ingres-admin@cariboulake.com] On Behalf Of Paul Mason
Sent: 20 January 2005 14:54
To: info-ingres@cariboulake.com
Subject: Re: [Info-ingres] Rollforwarddb disaster

On Thu, 20 Jan 2005 13:38:03 -0000, martin.bowes@ctsu.ox.ac.uk
<martin.bowes@ctsu.ox.ac.uk> wrote:

database

very

swap

There is no 'standard' config file swap technique, that's my point.
All config file manipulation should be through Ingres programs.

a

is

Jon

Actually I understood he copied the DMP cnf to the DATA area - the DMP
one having been recovered from backup.

a

do

It's possible such a technote exists. It may have been unavoidable in
older versions of Ingres and a technote written at that time may still
exist.

been in

just to

a

why

thought

I think you misunderstood me - the 'required copies' are done by
Ingres itself - no manually copying should be necessary.

Which requires you to recover the dmp, ckp and jnl directories from
your backup tape - it doesn't require any copying between dmp and
data.

restore

Often

have

You still have to have symlinks to appear as if you have the same
paths. So you'll have the appearance of the same directory structure
as your backup from which you can restore the necessary files then
rollforward.

I am willing to concede that it's useful to have a valid cnf in the
data directory because you need it to run infodb. An infodb
-file=<some cnf file> would be a useful enhancement. I used to have my
own program to do that but I didn't keep it updated. For checking the
paths and dbname  "strings aaaaaaaa.cnf" will work just as well.

to

I'd

Burning

Officially we *don't* support symlinks - the main reason being it's
too easy to make a mistake. However people use them and they work. But
they work at the operating system level and not at the Ingres level.
So we can't remove support in that sense. I suppose we could make a
change that meant they no longer work in certain circumstances but I
can't think of an example. All the file access stuff is in common
routines in CL and I doubt it's changed much in years.