The first sign of trouble was Tuesday last week, but we didn't realize how bad
it was until Thurday.  Check your calendars -- 9/11 and the 13th.  What could
possibly go wrong?

All is well now, including recovery of the data it ate when the problem first
appeared.  If that seems like a long time to recover, I spent about half my
time on other projects in between.  Also, I had to restore some files twice
because the Thursday problem was that we couldn't read the files I restored
Wednesday from an older backup tape to recover the data that was missing
Tuesday.

The missing data was right where the odds said it should be - in the biggest
data set of our biggest data base, the invoice detail.  Fortunately I have
utility programs to write the contents of a data base to & from flat files
(kind of souped-up Cobol versions of the old Fortran disk2db and db2disk).  So
when HP wanted me to purge the files so they could spare out the bad sectors,
I chased the users off, dumped the db to flat files, verified that the error
messages included the affected invoice numbers and a count of the missing
entries, purged the data base, and let HP do their thing, then refilled it and
let users back on.  Wednesday I restored the base from the previous backup to
a different account.  Thursday I found that copy was unreadable because of new
disk errors, so it was time to replace the drive and do the whole install
thing before I restored the data again.   Later, I slapped together a quick
version of the data-base dump program that would dump only the detail of our
26 bad invoices, from the other account.  Then the unmodified data-base
refiller added them into the real data base.
Even mickey-mouse data-base tools can to the job if you have source code and
can recompile for special projects.  I think this is only the 2nd or 3rd time
I have had to do special one-off versions since I wrote them back in the
mid-eighties.

* To join/leave the list, search archives, change list settings, *
* etc., please visit http://raven.utc.edu/archives/hp3000-l.html *