Tiffany:

You definitely need to have a test environment set up separate from
your production environment.  (Even then I've seen people screw up and
get fired because the front ends can look so similar).

If you have metalink access, type in "recovery scenarios" in the
knowledge browser.  Plenty of ideas scattered about there, especially
if you run across the scenarios they use in the classes (and note:
62385.1 is a pretty decent list of basic things to try).  Recovery is
_the_ most significant dba skill, and you are right to recognize that
you don't have valid backup procedures unless they've been tested.  You
need to know it cold for when you need it for real.

jg

Always, always, always test recoveries in a non-production environment.
It is a great idea to use a production backup to do so (kills two birds
with one stone). Need to refresh a development/test environment? How
about using the latest production backup?

Think about different recovery scenarios. Loss of a single table, loss
of a datafile, loss of a device, controller, system, data center, etc.
Work out how you would recover from each, how long it would take, how
much data would be lost, etc.

Presented for your consideration
"The responsibility of a DBA is not to back up the database...the
responsibility of the DBA is to recover the database!" (paraphrase of
Tim Gorman).

I recall a discussion at a user group meeting where a dba was telling
the story of a new tape drive in their backup system. Seems that there
was a slight miscalibration and the head would move a fraction of a
millimeter each time it wrote a new tape. Tapes would write
successfully, would be verified successfully...and could not ever be
read again!

I myself went through a situation where a bug caused the database to be
unrecoverable. Not fun!

Regards,
Daniel Fink

Companies normally have DR tests, just like fire drills. Typically, those
tests go from primary and standby databases switching places all the way
to going to remote location and restoring 1.1TB database at a spare
location which can be provided by companies like IBM or SunGuard. It is
important to understand that DR document should include the possibility of
failure of any component, like router, name server, firewall, application
server, web server, VPN server or LDAP server. Without the name server or
router your clients will not be able to find the database server, even if
the latter is perfectly operational.

The first thing to decide when writing a DR document is how far do you
want to go and what do you want to protect the company from? Are we
talking about major malfunction (like the 2003 big power outage) or total
loss of a location as after Katrina or 9/11? What is the cost of the
downtime and what are the time constraints? If there is an outage, how
long can the company afford to stay down? If the allowed downtime is long
enough, you can get away with restoring from backup in the moment of a
problem or simply activating a standby which can be located at the other
end of the same building.
The second thing to decide is how much data can you afford to lose? If it
is a dating database that keeps "compatibility points" and 10 million
addresses of ineligible and undesired bachelors, you can afford to lose
more data then if you are operating an on-line banking database. Last but
not least is how much does the company want to spend on data protection?
DR plans are frequently the first casualty of cost cutting. Suddenly,
non-technical people (CFO, for instance) bring in this wonderful sales guy
from EMC telling you that RAID-5 is just fine and is equally as fast as
RAID 1+0 or that those nasty old PA RISC boxes can easily be replaced by
nice new thingys with quad-Opteron motherboard and running RH. When you
find an unknown coffee bags in the kitchen in the place where Maxwell
House coffee bags used to be, it's time to get your resume on the Dice.
Maxwell House coffee is critical for DR plans as it enables your DBA to
perform database recovery at 03:30 AM. It's good to the last drop. CIO is
usually extremely touchy when it comes to signing checks, especially for
disaster recovery. Cheap solutions are abundant and normally do not work
without an extreme effort and many hours of pointless toiling for which
nobody will thank you. DR is primarily a business decision for which you
will need support of the senior management of the company.

Why am I telling you all this?
1) You are a junior DBA person and obviously are in charge of the
  company's DR plan. That is usually a grave mistake and a sign of the
  company trying to save money on the wrong thing. Nothing personal,
  but as a long time DBA, I don't think that a DR plan should be
  tasked to a junior person. That is the job for your team lead.
2) You don't have a DR drill. Fire drills are required by law, but
  not the DR drills. To be effective, DR drill must be performed at
  least once a year. A&E, the company that I had a consulting gig with
  for a few months, performed DR drills monthly. They have two locations
  (NYC and Stamford, CT) and they switch roles of the primary and standby
  databases, name servers, routers and they go through the whole 9 yards.
  Now that's the company conscious of the data security. You don't even
  know whether your backup can be restored. That leads me to the
  conclusion that RESTORE DATABASE VALIDATE is not a part of your weekly
  backup routine. (This RMAN command scans the last backup and checks
  whether it can be restored)

If your company is trying to save some money and if you, as a junior DBA
person, are in charge of DR plan, then you are what is commonly known as a
"scapegoat". Anything happens, and it's your job on the line. For a DBA,
an item like "I was fired when I lost company's production database"
doesn't look too nice on the resume and somewhat diminishes chances of
getting hired again as a DBA. Try clarifying the points mentioned above
with the management and if you don't get clear and satisfactory answers,
run like hell.

I humbly apologize if this sounds harsh and cynical, but I assure you,
it's a cold world out there. You are free to hate me if you so desire.

The Oracle class on Backup and Recovery was one of the best I have
taken.  Not only do you learn how to deal with nearly 20 different
scenerios the lab actually tests this knowledge.  Our instructor would
cause a failure and you would have to figure out which of the 20 it
was.  Good labs.

-----------------------------------------------------
Come race with us!
http://www.mgpmrc.org

That's called a black hole backup. It's essentially equivalent to
doing backup to /dev/null. On the plus side, you will never run
out of space, but you might have a problem with restoring it.

5 MONTHS? Are you sure that the tape is ripe enough for a recovery
test after only 5 months? You must let sun flares and electromagnetic
storms to take their natural course, as Simon Trevaglia would say on
certain occasions. Restoring 5 months old backup makes a lot of business
sense, I'm sure that numerous business analysts would be grateful to you
for providing them less then half a year old database, but I'd rather
let that tape to mature for few more months before attempting to verify it.

What is a "recovery scenario"?  DBA has to be able to restore the
database. DBA does backup of the database, DBA does backup of archivelogs,
DBA does a full export of RMAN catalog database afterward, preferably by
using scheduled scripts. Tapes then go to the tape duplicator where they
are duplicated and stored in two different locations. If and when the need
arises, DBA restores the database, from the backup that is less then 24
hours old, if possible, and if it isn't, then from the newest available
backup. Testing recovery is done from the complete set of tapes, that's
it. In case of a failure, DBA has to know what to restore and where. If
you have a RAC with one or two standby databases, then you don't have a
failure. You lose one, you continue with another. The same database is
still available. Emergency restore to another server is done only when
everything is lost.
DR test is not a database recovery course to practice recovering
tablespace containing the precious EMP and DEPT tables. DR test is a
simulation of a disaster in which, typically, the whole data center is
lost. In other words, someone tells you that all your base are belong to
us and that you are on the way to destruction.

I believe that management may have something to say about the partial
recovery. They might not be thrilled by missing data.

In addition to that, "BSE certification" was meant to bring a smile on
the faces of my Commonwealth friends, like Jonathan, Nuno and Niall. BSE
certification of a database would be especially popular type of request in
UK. DBA asking for such certification would have to be an Oracle
Certifiable Person, OCP for short.