IT Guy's Worst Nightmare

[Jack The Riffer]

Can you say FIRED????

http://www.msnbc.msn.com/id/17702021/

[AlexL]

Come on, this is America. You can't get fired for that.

The employee would have been given a "first warning". After that comes a second warning, and then a reviewal of their managers expectations for their role, and only following a reviewal one year later to prove that the written expectations weren't met could he be fired.


- E.

[Ralph E.]

My bad, I thought the answer was "any Microsoft product".

[Jason1212]

Why fire that guy? It's not his fault they have a crappy contingency plan and horrible data storage.

[Nuclear Vampire]

OMG. I just emailed that to all the guys at work. We've been bitching about getting new tapes and drives for over a year now. We've had so many issues of bad old tapes that can't be read coz they're too old and have been used too many times putting errors all over the place and locking up drives it's ridiculous. VMS is pretty gay that way. The new budget comes out soon and we all worked on a proposal for new drives and tapes, so hopefully we get them. You can complain all you want about "what if this worst case scenario happens", but until it actually does, the executive won't give it any serious thought. So frustrating

[UFORocks]

OMG. I just emailed that to all the guys at work. We've been bitching about getting new tapes and drives for over a year now. We've had so many issues of bad old tapes that can't be read coz they're too old and have been used too many times putting errors all over the place and locking up drives it's ridiculous. VMS is pretty gay that way. The new budget comes out soon and we all worked on a proposal for new drives and tapes, so hopefully we get them. You can complain all you want about "what if this worst case scenario happens", but until it actually does, the executive won't give it any serious thought. So frustrating

VMS is awful. Im so glad we finally got rid of it.

[Freyr]

My bad, I thought the answer was "any Microsoft product".

:ROTF: So true!

[buzzsaww]

In addition to backups you need to do regular restore testing and verification.

[bibz]

how brutal did he wipe that drive? surely with that much cashflow you could restore the data with those vacuum type dealies.

[UFORocks]

In addition to backups you need to do regular restore testing and verification.

Yeah, regularly scheduled disaster recovery is a must when you're dealing with billions of dollars.

Some of our customers are on double and tripple mirrors using very large san devices plus onsite and offsite tape backup. One such database we have is in the 7tb range, split 7 times.

how brutal did he wipe that drive? surely with that much cashflow you could restore the data with those vacuum type dealies.

From what I understand it is very costly and isnt very reliable, not for this type of situation.

[PowerTube]

how brutal did he wipe that drive? surely with that much cashflow you could restore the data with those vacuum type dealies.

The biggest problem with drive recovery is when the file allocation tables are gone. Sure, data will still be written to the platters, but good luck in making any sense of it.

[fett]

Only hedge funds lose that kind of money. The Alaska thing is just a glitch. Just wait until all this cyber-money stuff comes unglued. That "Will" be an issue.

[buzzsaww]

or wait till the fannie mae/freddie mac house of cards tumbles.

anyway I would have handed in my resignation if I had done such a bonehead thing. I can't fathom having one set of tapes.

[rjohnstone]

We use hot fail over systems for DR.
Data is synced in real-time to a mirror system, not drive, but a complete system.
Those DR systems are located in another state.
Plus the systems use mirrored drives and they are backed up to tape every night. Tapes are verified weekly.

All we have to do is replace the DNS host entries on our DNS servers with the DR addresses and we're back in business.
Well.. it's not that simple, but that covers about 80% of our business critical systems.

The others would require some additional work.


Should the guy be fired... I don't think so. I doubt it was malicious.
Shit happens.

[fett]

It takes a master to build a house of cards 5 feet tall. It only takes a sneeze from one player/watcher to have it come tumbling down. It's happened before and it will happen again. "Cash is King"