Symetrik Domino Consulting

Today just as I got into the office and started my daily grind, I noticed that my mail server had hung (ok, well not after I checked to see if 8.01 was out yet…as far as I know it will only be a few more hours).

So, then I stood up and told my co-workers who were starting to notice the same thing.

I then noticed it wasn’t just one server, it was all of our servers. Everyone started rushing around, and trying to figure out what to do.

You see we have just outsourced all of our servers to an externalized hosting facility two weeks ago. Last week, we had a similar outage from 12AM to 7:15AM, no patrol monitoring alerts, nothing. We had to rely on users in Brazil and America to let us know about the outage 4 hours after it was in effect. Our Inbound/Outbound Internet gateways, our central servers that act as relays for all internal mail (by design) and our POP3, Passthru, and Domino servers that serve all users who sit in sites that are not big enough to have their own server.

Quite alot of stuff to be going down for worldwide operations.

Today, it turns out that the SAN went down, so all servers that were connected to the SAN at the hosting facility were down.

Fine and dandy, especially since the reason we went with these guys is to manage out data, ensure our backups, and provide a Disaster Recovery mirror facility. It’s been almost 3 hours now and just cutting over to the disaster recover center hasn’t been successful.

The worst part? The DHCP servers were connected to the SAN, so anyone who came in late today, rebooted their machine, and turned a machine on after the DHCP server cannot get an IP address and utilize Gmail to conduct business (or carelessly surf the Internet and post to their blog).