04/23/2006 Unreal server outage

Before I begin, sorry kids, this is not an April fools joke

Sometime during the night, our co-lo (again) lost a ups battery.

This happened just at the wrong moment and smoked the current RPG data on the server. Unfortunately, those changes were also replicated to the backup server.

As the Hitchhikers Guide to the galaxy has taught us, DON'T PANIC!

We have a full backup of the RPG data from 2/22/06. This will be the base set that we make any changes against.

I've been debating how to recover more data all day; however Llachior came up with the best idea.

**Updated
For the next 15 days (starting tomorrow), we'll multiply the exp gained in a game by 2x, (including medic). Also, I'll guarantee the Flak +5 and RL +5 wont disappear during the next month. (Thanks for the idea Llach)

**New
Also, for the next 40 days, I've reduced the requirements to attain level 36 by more than half.

This means that a lower level character can effectively level up in 4x the time for the next 11 days, and then 2x the time for the followin 29 days.

We have a backup of all players level data who were level 36 or higher as of 3/25/06. We will modify the players RPG data to reflect that updated base level.

The unreal server will likely be down throughout the end of the day. Shantara and I will be working furiously to integrate the level information we have from the 25th there's some 200+ players to work through, and we do both have day jobs, therefore, we believe it's going to take the remainder of the day integrate that data.

Active reserved slot players will have their slots bumped up by one week to compensate for this, as best as we can.

If you feel that you are in any way being cheated by this strategy, you are welcome to email administrators@disastrousconsequences.com with what you feel would be honest and fair for your case, and we will do our best to accommodate you.

To prevent this in the future, we're going to strengthen the backup strategy to create a copy of the RPG data to an additional backup on a week by week basis, so if the core file gets corrupted we'll still have a more recent backup than a month prior.

I suppose the silver lining on all this is that the we're fortunate that no hardware was damaged, and that the backup from the end of Feb was intact.

Once again, I'd like to express my sincerest apologies on this,

TheDruidXpawX

It may have had something to do with the April RPG thing, though I am just guessing since that was the newest thing prior to the corruption.

Its gonna take me forever to get from lvl 32 to lvl 34 again.

Yes, I suck at the moment. haha

Sounds like a suckbag situation, hope you two aren't up all night.

macadamia wrote:

It may have had something to do with the April RPG thing, though I am just guessing since that was the newest thing prior to the corruption.

Nope. It's directly related to a UPS battery in the co-location facility going bad at the wrong time and the power to the server going off.

Thats the 2nd time in a year CSol has fragged a UPS isn't it? Bummer Dru, hope you guys can swing it, sounds like a real fair solution.

Thats a real bummerr..... O well just gotta spend more time with you guys

I geuss we game too hard for it

I can tell you from personal experience that UPS death is mostly unpredictable - except that if you buy all the batteries/UPS at the same time, you'll find that they go in waves.

It used to be (like, say, 10 years ago) that, at least with Unix boxen, you didn't want to reboot or shut them down if you didn't have to ... that's when you found out that something was really broken. Particularly with the drives and file systems, a power interruption could mean the death of the box. It's not so much the case now with hardware in general, and it's more rare now to have a file system loss ... things are pretty much more stable through unexpected outages ... except for UPS.

I'd say roughly 70% of mine, if they fail a self test, fail very abruptly - the self test causes the attached machine to suffer a power interruption. And even the self tests don't always warn you ... you have a power outage and your UPS (which tested okay a week ago) ends up with a bad battery.

So, in short, don't anyone go blaming Dru or Shan for this - they're doing more than we do on the Mud I host. And really, you can't blame the hoster either - UPS are just one of those things that if you're going to have bad luck, that's where you're going to have it.

Besides ... my tendonitis was acting up anyway.

Ah, bummer.
Just came back from vacation and tried to go on, but the server was down.. got all worried. Thanks Shan and Dru for your work on this, really appreciated. And the 2x solution - great

I like the solution. Appreciate all the hard work you two put in to maintain the server and the hard work you will be putting in to bring it back up. But, I do have an essay I have to finish up, so I won't be dissapointed if the server is down till tommorow.

I knew it! The nalimers (nalis with deemers) did it!!!

I'm sorry this happened Dru and Shan, and I truly feel your pain (I deal with this type of issue at work as well).

Crap happens. You give us this great server to play on. We can hardly complain if something fails. How big is the database? Perhaps we can arrange some offsite Internet backups for you.

that b cool by me love u 2 kids

ARG!

The X2 idea is good, but Druid, February 22 Through April 3. This is 40 days (and probably gonna be more). Do you think you could give us 40+ days of X2?

Not counting that its gonna be impossible for some of us to play as much as in winter.
Especially when the project im on ask me to do overtime on weekends

No sweat Dru and Shan! It happens, we'll all live, don't go spaztic trying to fix it in a hurry and get yourselves all nerved up and prob sick again.

By the way can we help on something? Parsing data or text files etc? Ill be glad to give it some hours.