This thread reminded me of a post to the famously friendly
[log in to unmask] list, at least at the point at which I understood
better the extreme nature of the situation John was planning for. I
eventually emailed the author, to thank him for his post, which I still
enjoy reading. I assume that there is some art and exaggeration in the
events described, and possible several events have been combined for effect.
For the record, there are apparently some serious concerns with halon, just
as there were with what halon replaced...
Greg Stigers
http://www.cgiusa.com
-----Original Message-----
If my first two weeks at a new job have taught me anything, it has
been "Never plan to test anything."
I planned to test the backup system as my first serious project. It's a
4TB DLT changer backing up a bunch of database servers and a a couple EMC
arrays. It had been set up by a consultant two years ago, and noone had
ever even tried to do a restore. My bad. The day I decided to do this, a
disk died on one of the database machines. Eh, one out of two logical
volumes isn't bad. At least I got the important one back, and discovered
that the other one was just being backed up as an empty mount point. At
least I know which parts of the backup system need more spackle now.
This, however, was just a prelude to the true destructive force of asking
about emergency plans, as I discovered at quarter to 5 on the day I had
asked about them. How did the building UPS work? (poorly) Is the
cooling system for the machine platform on the UPS? (no) How many things
start beeping/screaming/crying when the server room breaks 85 degrees?
(More than 10.) What happens when our power provider has a wierd blowout
at the substation and starts sending our building gimped power? (Pulsing
brownout, sparks, flame, lots of smoke, noise and flashing lights. Men
with axes storm the building.) What happens when they cut in the big
diesel generator outside as the building UPS gets low? (Don't even ask.
It's not a hot switch.) Why are there guys scrambling to run giant power
cables from the diesel generator outside in through the basement window?
(I told you not to ask.) What happens when the building telco guys are in
the middle of a huge noninteruptable upgrade and the building power goes
out in a cloud of electrical smoke, and farther out in a cloud of diesel
smoke? (We laugh. They cry.) Will the big HP-UX machines running
Peoplesoft that I'm now in charge of but have never seen boot come up
cleanly, even after a nice clean shutdown? (Hell no.) Why did the main
database server come up with CPU 01 3? Where did 2 go? (Beats the hell
out of me.) Why is the production database console-spewing Fibre Channel
errors and not booting anymore? (Oh, it just does that, don't worry. You
might need to power cycle the server a few times if it hangs in the middle
of booting and get a goat out of the back room for it. Ask to borrow an
axe.) What's that smell? (Burning plastic, toxic fire extinguisher smog,
diesel, salami, geek.) Why does someone keep running to the controls for
the halon system every time it starts beeping? (You should have learned
to not ask by now.) Do I get overtime? (Yes, and that makes it all
worthwhile.)
The best part is, this used to happen all the time. It's apparently a lot
better now. ;)
Goodnight, I'm going to bed.
(Note: Possible slight delusions. I'm tired. Don't hold me to any of
this. ;)
Chuck McKenzie
[log in to unmask]
* To join/leave the list, search archives, change list settings, *
* etc., please visit http://raven.utc.edu/archives/hp3000-l.html *
|
