Jay Kae Posted August 3, 2013 Share Posted August 3, 2013 This story begins about 60 hours ago when EIG went to go and do some normal maintenance on their datacenter, during this maintenance something happened that was not part of the plan, I will not go into detail on what that was but let's just say, it was not good. Errors started occurring on websites that should not occur, all services I run are via EIG but apart from my 'hobby' services also a plethora for my work life. Let me just say that Orbx, REX and OZx were the least of my problems in the last 60 hours. In this case there was nothing I could have done to avoid the downtime as it was not my server that was down but the whole datacenter that had bitten the dust and everything went south. I can only apologise for the inconvenience this has caused. No matter what contingency plan I would and could have had in place, this was a one in a billion chance of happening and ... it happened. Link to comment Share on other sites More sharing options...
Iain Emms Posted August 3, 2013 Share Posted August 3, 2013 No worries Jay, and I thank you for the hard work you do here on our behalf all the time, it seems to working alright here now. cheers Iain Link to comment Share on other sites More sharing options...
JanTenner Posted August 3, 2013 Share Posted August 3, 2013 No need to worry and also not for an apology. Everyone appreciates your hard work. Things can go wrong sometimes without anybody makin a mistake. So one more time thanks for your hard work here Link to comment Share on other sites More sharing options...
Stillwater Posted August 3, 2013 Share Posted August 3, 2013 Jay, have a good rest over the weekend! Well deserved. Link to comment Share on other sites More sharing options...
Ripcord Posted August 3, 2013 Share Posted August 3, 2013 JK no harm done here. Far worse happens here on my end, dealing the local cable/broadband provider, trust me. And we appreciate your efforts in looking after us. Link to comment Share on other sites More sharing options...
fly37 Posted August 3, 2013 Share Posted August 3, 2013 No need to worry.. it is not terrible and I understand.. and lot understand like me.. Thank for all you are doing! Link to comment Share on other sites More sharing options...
WilliamJSS Posted August 3, 2013 Share Posted August 3, 2013 And this....is why we are not allowed to have coffee in our server room here at work.....lol Link to comment Share on other sites More sharing options...
Wolter van der Spoel Posted August 3, 2013 Share Posted August 3, 2013 No matter what contingency plan I would and could have had in place, this was a one in a billion chance of happening and ... it happened. Jeeeeeeeeeeeeeez ! I wish we all would be that lucky with a lottery No worries Jay, thank you for your tireless efforts keeping us up and running Link to comment Share on other sites More sharing options...
ramonb Posted August 3, 2013 Share Posted August 3, 2013 Hello Jay Kae Thank you for the explanation, from what I read it was a "global" problem.I think all of us here completely understand that things happen! No need to apologize, really. Thank you for your hard work on keeping so many sites working like a charm. BTW: I never heard of RailGun before. But after reading about it, we are looking into it and will probably implement it on one of our Telemedicine Solutions. Cheers. Link to comment Share on other sites More sharing options...
spirit_66 Posted August 3, 2013 Share Posted August 3, 2013 Jay, don't worry because everything is OK. Thanks for your work. Spirit Link to comment Share on other sites More sharing options...
momo737 Posted August 3, 2013 Share Posted August 3, 2013 I smell the man in black conspiracy!! Link to comment Share on other sites More sharing options...
Wolf2grey Posted August 4, 2013 Share Posted August 4, 2013 Working fine now bro.......Hell I could do with a break from ORBX Forums.....lol.....I am on it WAY too much at work...... Link to comment Share on other sites More sharing options...
Mallard Posted August 4, 2013 Share Posted August 4, 2013 No need at all for you to apologise, Jay Kae! And things appear to be up and running again... Cheers Mallard Link to comment Share on other sites More sharing options...
moonman Posted August 4, 2013 Share Posted August 4, 2013 Indeed Jay KAe - no need to apologize at all - in fact: Great to read that nothing "serious" happened! Cheers, Christoph Link to comment Share on other sites More sharing options...
Jay Kae Posted August 4, 2013 Author Share Posted August 4, 2013 Thanks guys, it was a comedy of errors really but I suppose the good thing now is that they have put things in place to avoid this issue from happening in the future Link to comment Share on other sites More sharing options...
RLSinCO Posted August 6, 2013 Share Posted August 6, 2013 Reminds me of a major outage we experienced a few years ago in one of our (co-hosted) tier 3 global data centers in the UK (not going to name them but let's just say a MAJOR provider). They were performing maintenance when the entire data center lost power. That's not supposed to happen, in fact we were guaranteed that could never happen. Battery back up did not come on, and because their generator system monitored battery backups for runtime in order to kick off the generators and didn't see a battery drain...no generator power either. Darkness. Turned out to be a part that cost no more than 5 pounds, no kidding. One in a billion chance...needless to say, they were supremely embarrassed. Jay Kae - I remember what you did while one of our bretheren's forums was under a DOS attack and you literally stayed up 3 days straight fighting it off...yeah - no need for you to apologize mate. In fact - if you're ever looking for a new job PM me; I manage global IT operations worldwide and will get you in our doors pronto! Link to comment Share on other sites More sharing options...
Jay Kae Posted August 6, 2013 Author Share Posted August 6, 2013 Thank you RLSinCo I will keep it in mind, I cannot believe you remember that, I was destroyed after that stint lol Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.