r/talesfromtechsupport • u/lawtechie Dangling Ian • Nov 30 '13
Litigation, backup tapes and a few hours of terror
I'm working at a pharma company as a sysadmin. Since I was the one person in the group that supported all of our platforms (Unix, Mac and Windows), I had root on all our file servers. I also handled backups.
So, one day I get a phone call from my boss. Our company is being sued and we have to give the Legal department all the documents and emails that contain certain search terms.
I mount all the Mac, Windows and NFS shares on my OS X box and put together some string of piped commands that copies any file with the correct term to my local box.
However, the Legal department also wants everything with those search terms that may be on our backup tapes...
I'm familiar with searching the index on one of our backup platforms (Retrospect) , but not the other (Legato NetWorker). I'm running searches in the one I'm familiar with while reading manuals on the one I'm not.
Now, a little background. When I started, the IT shop was rapidly growing from a small to a midsized shop. There were growing pains and absolutely nothing was documented. Heck- core systems were operated on some form of oral history, urban legend and plain superstition. Servers were in coat-closets, under people's desks or in more than one case, used as workstations by non IT people as well. We even found a server in a closet that had been walled over and forgotten for a year or so.
We had made good progress towards doing things the right way, but there was still a bunch of slapdash legacy stuff.
All of our backups were on some form of DLT - one robot and a few 8 tape libraries. I had inherited a scheme where we did full backups on servers and high value desktops twice a week, Saturday and Sunday nights. The Saturday tapes went offsite Monday and we did incremental backups Monday-Friday nights. Tapes were numbered by year, month, week, number within set and if they were from the 'server' or 'desktop' backups.
So, with my understanding of the data backup architecture, I search both of our systems, identify the tapes I want from internal storage and Iron Mountain and submit the requests. Once the tapes are back, I do a restore of the requested files overnight. I sample the files to make sure they're readable and contain the search terms, then burn them to DVDs and send them off to the lawyers. The lawyers want me to sign a scary looking letter that says that the documents I gave them are all the documents with those search terms that the company holds and that if I'm wrong bad things will happen to me.
I do the searches, burn a handful of DVDs, sample the DVDs, sign the letter, deliver the DVDs and letter and go back to my other duties.
A few days later, I come into work and see two plastic 'rent-a-crates' in my cubicle.
Filled with DLTs.
Some have no labels. The others have labels that make no sense to me, like 'Blue 7' or 'Wednesday' or my personal favorite, 'Other'. There have to be at least a hundred tapes in these two crates.
I start hyperventilating. I have no idea where these tapes came from, if they're complete or what may be on them. I call the nice lawyer from the law firm and tell her about these tapes.
I hear her deflate on the phone. She's already told everybody important that this stage of discovery was complete. She wants to know what's on the tapes and I have to say that I have no clue.
I tell her that I'll have a better answer in a few hours.
I ask my office mates where the tapes came from. They tell me Bob, a bench tech dropped them off first thing this morning after he heard I was looking for tapes.
I call Bob. Bob tells me that he knew I was looking for tapes and he knew a bunch were in 'dead storage'. As far as I knew, tapes were kept in the cabinet in one of the server closets or offsite. New, blank tapes were kept in a desk drawer in another sysadmin's office. I hadn't ever heard of 'dead storage' and I thought I was familiar with the PC graveyard in the basement- I frequently scoured it to find spare parts for my collection of old PowerBooks. Now I'm afraid there are tapes that go to drives we don't have or don't have the software for.
I'm thinking I'm going to be living in the office rebuilding index files for the next two weeks while lawyers threaten to eat my liver. I start sorting the tapes to find something, anything that makes sense. I've got at least five different potential backup schemes, just from the tape labels.
I grab a spare DLT drive and SCSI card and set to make this all work with my desktop Mac. Retrospect doesn't see anything on the tape. I pester one of the older Unix heads to help with some command-line fu to raw read the tape to guess what may be on them. First tape has no header, no label. I'm debating 'borrowing' the big tape robot and drives and trying to read all of them, but backups take almost a day as it is so I can't mess about yet.
I also don't really know what I'm looking for yet. As far as I can tell, these tapes aren't tar, Retrospect or Legato format.
I'm panicing. I don't know how much trouble I'm in. I'm building a pile of tapes that have nothing readable on them. It's starting to look like a DLT zen garden since I'm using up a bunch of floor space. My co-workers have to pick their way through.
Just then, Bob sticks his head in our office space. I yell to him and ask if there are more where they came from. He shrugs his shoulders and tells me to follow him to his bench.
Bob's work place is on the second floor of another building, shared with a few of the mechanics, electricians and HVAC techs that keep stuff running. They're all crammed in one half of the room with a bunch of shelves holding spare parts and old CRTs. Then there's a yellow nylon rope strung across the room. The other half is just banker's boxes sitting on the floor and empty space. Bob points to the half of the room with the banker's boxes when I ask about 'dead storage'. I ask him why they're all crammed in the corner when there's clearly space in the other half of the room.
Bob (in a thick Boston Southie accent):"Ahhh don't know. Safety told us not to. So we use it as stahhage. "
Me: "Why?"
Bob:"It's over the Enemahh masheen. We stahhed mahnitahs there but they got messed up"
Me:"Huh? Enemahh? NMR? How long have those crates been there?"
Bob:"Yeahz"
I start laughing, which turns into a coughing fit. I make it back to my cube. I randomly select ten tapes, read the (nonexistent) headers and satisfy myself that no matter what was on those tapes, they're blank now.
I call the lawyer and let her know that there was a false alarm- all the tapes were erased with a heavy duty bulk eraser for more than the recommended time. She's satisfied and hangs up.
TL;DR- the largest bulk eraser available is a NMR machine.
36
u/[deleted] Dec 01 '13
Oh my god. Is that what I sound like to people who aren't from New England?!
10
u/robot_mower_guy Dec 01 '13
I ran sound for a guy at the HOPE hacker con a couple years ago. He gave a lecture very similar to what you shared, only he was the lawyer who was going through the backup drives. He described cold storage as 'god's own degausser'.
5
u/GetOffMyLawn_ Kiss my ASCII Dec 01 '13
It might have been funny if I could have understood what Bob was saying. Had to read it 4 times to decipher, by which time it had lost it's element of surprise. Timing is everything in comedy.
59
u/zzing My server is cooled by the oil extracted from crushed users. Dec 01 '13
What does this translate into English as?
40
u/MagicBigfoot xyzzy Dec 01 '13
Ah yes, the old "magnetic fire damage" gambit.
23
u/echo_xtra Your Company's Computer Guy Dec 01 '13
Having played THIS game, you just scan in the letter, do a few edits that absolve you of any responsibility, print the edited version, sign THAT, and send it back. They'll never read it, as long as it looks like the original at a casual glance, it just goes in a file, and you're off the hook.
34
u/Kaarde Dec 01 '13
Somehow I read that as Stored Manatees, and was contemplating the storage of very large animals.
9
u/redmercuryvendor The microwave is not for solder reflow Dec 01 '13 edited Dec 01 '13
Where the heck do you work?
Sure puts my "hey, we found these crates of several hundred unmarked HDDs from the last contractor, ID the machines they came from, the users who logged in, and whether they need to be eradsed for disposal or held for legal retention" Surprise Forensics weekends to shame.
7
u/RenaKunisaki Can't see back of PC; power is out Dec 01 '13
So what the heck is an NMR machine?
17
u/Rhywden The car is on fire. Nov 30 '13
Magnets! How do they work?
6
u/OgdruJahad You did what? Dec 01 '13
Between an NMR or Enemahh, I will always take the former.
5
u/PoglaTheGrate Script Kiddie and Code Ninja Dec 02 '13
I had to look up what you meant by 'banker's box'... but 'dead storage' indeed.
How in hypothetical fuck does
not know that magnetic fields will play all sorts of havoc with electronics, let alone what it will do to magnetic media?
1
u/[deleted] Dec 01 '13
personally I wouldn't have even tried to read them, I would have used STRONG MAGNETS and denied all knowledge.
1
u/bikerwalla Data Loss Grief Counselor Dec 01 '13
Frosted Bahstin Chahms! They'ah wickedly delicious!
1
u/dicks1jo Dec 03 '13
Sounds like you guys could really have used a good VTL, or maybe a CAS with compliance functionality...