I work in an organization that uses Lotus Notes 6.5.x only for email. We are a commodities trading company with 2800 mail users in about 40 countries. We do not have a document management system in place, and many of our users send scanned contracts back and forth with other companies, which drives up the size of their mailboxes enormously. We currently have 1950 mailboxes that over 2GB (included replicas on hubs and cluster mates).

This is my first post on a series of posts that I will write throughout the course of the next 18 months on migrating from Lotus Domino 6.5.6 FP3 HF105 to Microsoft Exchange 2007.

To commemorate this first post, I have to refer to this link I stumbled on today from a slashdot article about the US State Department’s recent Email interruption. The post is called “Bedlam DL3, Me Too”. Very funny, and please note it is 2004. Don’t hate on MS too easily.

The actual project is to implement a global Active Directory and then implement Exchange 2007, and it is being managed by the new CIO, and a couple of other guys within our organization. I haven’t been involved with alot of the negotiations and initial planning with Microsoft.

Just before Christmas, I was called into a meeting to review the messaging side of the project for the final contract since one of the aforementioned guys was on holiday. The technical consultant on the Microsoft side was trying to push through it as fast a possible to get it done and signed. We, on the other hand were going through it with a fine toothed comb. Those guys were pretty sneaky, and took the general attitude of “Don’t worry, we’ll do it, it’ll all be covered, just trust me.” Worse than a used car salesman in alot of ways.

One of the main things that I noted is that they are going to use a tool to migrate existing mailboxes / contacts only for VIP users. Because the tool has to be licensed for each mailbox and each use. This tool apparently is very expensive to license. More details later on which product it is. I mentioned a tool that we use to convert outlook contacts to Lotus Notes which I happen to know works the other way and there started to be some back pedal.

Last week, I started to view some of the Exchange 2007 videos that are out on Microsoft here. For the most part, I can see that there are alot of elements of Outlook that are borrowed from Lotus Notes. On the other hand, there are some features, such as the out of office feature that blow Lotus Notes away. Let’s face it, if you manage a Lotus Notes infrastructure and only allow your users editor access to their mail file, out of office issues cause unnecessary user frustration, help desk calls, and administration effort.

Last week, I sat in on a product and contract review with a certain vendor that I’ll mention in a later post. They will handle the E-Discovery and archiving of exchange. This vendor is already our main backup vendor in most of our sites.

They started to explain how their product works, which utilizes Exchanges built-in journaling functionality to journal all messages to a given mailbox (I hope we can configure more than one) for journaling and E-Discovery. Normal mail archiving is handled by their product. The more they started to explain, the more we realized how much more hardware and more importantly how many SQL server licenses we will require, I think our team started to realize just how this project is going to go. We are already aware of the Microsoft marketing machine promises versus how reality plays out, and we are in the mindset of protecting ourselves from that, but at the end of the 18 month project, I wonder where we will be at (plus or minus) with the projected hardware and the projected annual cost (hardware, network, and server/user licenses).

Microsoft is designing the whole infrastructure and they are bringing in a team to implement it for us with our oversight. That’s how management wanted to handle it. There are pluses to this arrangement and there are minuses. I will report on both.

Most of our servers are clustered, and we use clustering for fail over, not load balancing, so our odd number servers are primary, and even number servers are the failover cluster mates.

Since at our hosted facility, the production environment was down, and the disaster recovery site wasn’t being backed up, we wanted to fall back the even number servers to our original facility in-house. This way, at least we would have a backup, or fall back plan, if our servers went down again at the outsource facility.

First, we removed the machines from the cluster in the NAB.

What we did next was to copy names.nsf, (our secondary address book), admin4.nsf, dc.nsf, da.nsf, and events.nsf to a USB disk on our local admin machine. We probably could have gotten away with just names.nsf, admin4.nsf, and dc.nsf, but just to be sure, we copied the others. We also probably could have copied domcfg.nsf and any resource reservations database to be very very sure, but we didn’t this time.

We also connected to the remote (outsourced) odd number machines and copied the notes.ini files in case there were any changes in the 3 weeks that we had cut-over to the host provider, then we configured the machines with a different IP address other than production and had them powered down.

Then we went to the physical old servers (in house), turned them on, making sure that the IP addresses were configured as the production IP addresses, copied the administrative databases.

copied the old notes.ini files to the USB stick, and brought them back to our admin desk. We then compared the notes.ini files of the in-house, with the outsourced, noted any changes, and copied the updated notes.ini to our in-house machines.

After a couple of tricks on the networking side of things to make sure that there wasn’t any problem in physically connecting to the old, in-house machines, we powered them up just as they were.

We manually replicated from the even number machines to the odd number machines before adding them back to the cluster together…but the way we chose to do this was to issue seperate replication commands by directory. This way we could get each directory replicating at the same time with multiple replicators since these servers have 8 processors in them, we could realistically get 8 replicators going without any problem (and probably more).

example:

> load replica serverodd/domain mail
> load replica serverodd/domain apps
> load replica serverodd/domain acct
> load replica serverodd/domain marketing

This way replication does not step on itself either. If you just type “load replica” eight times, the replicators will eventually get stuck all trying to replicate the same database and actually slow things down a bit.

Since the network was still connected, and we had a fast pipe to the outsource facility, and the subnets that both server rooms sit on…the replication started to catch up pretty quickly.

We had to catch replication up from 3 weeks of changes. The old servers that remained in-house were turned off after the cut-over. So any changes and updates that were made were not included on the even number servers which are in the in-house facility, but of course they existed on the odd servers that had been live and in production at the outsource facility.

Overall, it took us 2 to 3 hours (not including some quick planning to try and think through any “gotchas”) to undo a year of project planning to move our servers to an outsource facility.

This 2 to 3 hours meant rolling back 4 busy Domino servers back home.

We performe this operation from 3-7AM, so by 9AM when the business day started, the replication had caught up.

We did have a few minor problems, such as new users that had been created, archives that had been created that existed now on the odd, but not the even. The only small down side to all of this is that when we create archives, we don’t always put them on both cluster mates to save space, so there were a few archives that were created on the even number cluster mate at the outsourcing facility, but not the odd number cluster mate, so we couldn’t replicate them manually from the odd number server to the even number server (inhouse).

The solution to this problem was to do a database backup restore from the even number server backup image to the odd number server at the outsource facility, then we could create a replica on the even number server in house.

A week later and the outsource facility still does not know why all our our Domino servers went down at the same time except for the fact that it was SAN related and are looking into it.

Through all this, I’ve created this ideajam.net idea:

Create a way to synch the existence of databases between cluster members

So, then I stood up and told my co-workers who were starting to notice the same thing.

I then noticed it wasn’t just one server, it was all of our servers. Everyone started rushing around, and trying to figure out what to do.

You see we have just outsourced all of our servers to an externalized hosting facility two weeks ago. Last week, we had a similar outage from 12AM to 7:15AM, no patrol monitoring alerts, nothing. We had to rely on users in Brazil and America to let us know about the outage 4 hours after it was in effect. Our Inbound/Outbound Internet gateways, our central servers that act as relays for all internal mail (by design) and our POP3, Passthru, and Domino servers that serve all users who sit in sites that are not big enough to have their own server.

Quite alot of stuff to be going down for worldwide operations.

Today, it turns out that the SAN went down, so all servers that were connected to the SAN at the hosting facility were down.

Fine and dandy, especially since the reason we went with these guys is to manage out data, ensure our backups, and provide a Disaster Recovery mirror facility. It’s been almost 3 hours now and just cutting over to the disaster recover center hasn’t been successful.

The worst part? The DHCP servers were connected to the SAN, so anyone who came in late today, rebooted their machine, and turned a machine on after the DHCP server cannot get an IP address and utilize Gmail to conduct business (or carelessly surf the Internet and post to their blog).