When it rains, it pours...and gets really really messy

Just had an interesting evening here in Da'Burgh last night. I had no more than started eating my dinner when I got a phone call from work. The rains we were having were so severe, we had water flooding into the 3rd floor hallway where we have a small computer room. Now mind you, the building we are talking about has an entrance on one side on the 1st floor, but the building is on the side of a hill, and main entrance is on the other side from the 5th floor, with a parking lot entrance from the 4th (as well as the bridge to the building across the ravine from that floor). When I got there, I found water streaming in from under the door, and going around to the stairwell on the other side of the door, there was more than a foot of water there, at the bottom of that stairwell, and about 50 GPM pouring in under the door to the parking lot and down into the stairwell. The end result was about 3-5" under the raised floor in the hallway, with water going into our small computer room, as well as into the much larger computer room for the school of computer science. And up on the 4th floor, we had water pouring through the ceiling, and down off the lights, etc. But wait... there is more!

After shutting down that computer room, and helping the SCS folks figure out their risk, I got another call... another one of our computer rooms was overheating. But, try as I might, the keycard system at some point in the last 3-6 months had lost my authorization to a doorway I needed to get through. And so, I let the folks who had called about that room know that our servers were fair game as well. But does it end here? I bet that knowing me, you said "No", and if you did, you are right.

Now, our main data center is in a building on the edge of the Carnegie Mellon campus, and like many of the buildings, you can find 4-5 stories worth of elevation difference from one side of a building to another. For our building, this means that from the ground level, we have 4 levels of parking and a lower level, thanks to the hollow of which the building is on the edge. And down in the lower level, at the same level as the chilled water pumps, etc. in the neighbouring facilities building, are the main transformer rooms and the main electrial room. Our data center is designed with a redundant utility feed with an almost 800KVA capacity (we represent the majority of the building power demand). Well, the lower level of the garage flooded, with water being 2-3 ft deep in the electrical room. As a result, while we had not tripped the breakers carrying the 115KVA we were pulling at the time, we had to shutdown the main data center as well. If you want to see what things looked like, including the flooded loading dock which has a car hidden beneath the water, debris and dumpster, here is a URL to the few pics I managed to take. Also included is a link to the local paper with video of flooding elsewhere in town.

http://www.ka8zrt.com/photo_albums/cmu_flood_2009Jun17/index.html

http://www.post-gazette.com/multimedia/?videoID=102011

Gonna be a long few days at work.

You need some professional help.

No, I did not mean you, I meant the university.

You need to convince who ever has the authority to call in an accredited water damage restoration company, probably a large one from out of town (all the ones in town have contracts and will be busy).

There are one or 2 big companies that specialize in data and computers but honestly its very time intensive work.

I worked in the industry (water/fire damage restoration) for about 7 or 8 years. Get some professional help and things will go much faster.

oh, I did not say before, but your situation sucks, I have been there, kind of, twice.

I got into the restoration biz because I was interested the second time we lost our home, the first was when I was 16, heating tape started a fire when no one was home, in the winter. We lived far enough out that the fire trucks arrived about the time that the coals were about right for BBQ.

Second time was water, they were doing sewage maintenance and managed to fill our home, the house was not a total loss, but we did not live in it for 4 months.

Best lesson of my life: Insurance is expensive when you don't need it, and very cheap when you should have bought it.

University probably is doing a restoration company or that sort of deal. Our one computer room is being handled by the building manager for that building, and the garage/electrical area is being handled by the building manager for that building (which is actually not fully owned by the university, but was a consortium which includes the university).

Gah! I feel your pain - been there done that. Good luck!

ugh!!! that sucks!!! im sorry!!!