Domino @ Symetrik Design

We are considering virtualizing some of our Domino clusters. We don’t want to install Domino partition servers on the same box, but we want to virtual some of the clusters so that we have virtual machines on different physical hardware that are virtualized along with other virtual server (maybe non Domino).

We want to use VMWare.

What is the experience with VMWare and running production Domino mail servers? These servers will be connecting to a SAN for disk storage.

We previously had tried to virtual Domino at our host provider Atos Origin, but in the end had problems with it so we went with a white box solution (1 physical server for 1 Domino server). That was over a year ago though, so the problems with doing so may not exist anymore.

Looking forward to your experience and stories regarding the issue.

Last week, I met with Microsoft regarding one of our sites that is not currently on Domino. We will call this Site A. We built 2 clustered Domino servers for that site in September of last year and put it on hold because we didn’t want to go through the labor/cost of migrating 300 existing Outlook clients to Lotus Notes if we were only going to migrate them back to Outlook a few months later.

We’ve been waiting on how the contract negotiations with Microsoft will go and then determine through the Microsoft design, planning, and scheduling to see how long it will take.

The problem is that this site relies on a 3rd party host provider. We have no guarantee of backups, no resolve if the company goes bankrupt, no compliancy capturing, and then the added cost of paying for the external service.

Their existing mail setup is Outlook 2003 or 2007 accessing POP3 mailboxes with local .pst files. The meeting was to determine how we should proceed in the next few weeks because the management at that site is now very concerned several months after we had planned on bringing this service in-house that data is not secure and protected.
Read the rest of this entry »

This meeting took place on January 23rd, 2009 and I’m just getting to editing it so that it can be published.

These are meeting notes and general factoids that I jotted down. This was taken from a planning meeting with the Microsoft Senior Technical Account Manager who is handling the project for us and the Senior technical Engineer who is planning the environment.

The Microsoft Senior Technical Account Manager told me that quote:
Exchange has 60% of the email market worldwide.
20% lotus notes
10% Linux or “something”

The overall Active directory presentation and design was pretty good. It makes sense and the MS guys seemed knowledgeable and understood what we wanted and I felt like when they said they could accomodate what we want that they meant it.

Currently, we have 2 clustered Domino mail servers in 17 sites.
3 Blackberry servers in hub locations.
1 Pair of clustered hub servers (outbound SMTP gateways)
1 pair of clustered inbound SMTP gateway servers.
TOTAL: 41 Domino servers

MS Exchange design:
10 sites will have (2 MBX (CCR) + 2 CAS/HUB + 2 ISA)
4 hub2 sites will have (2 MBX (CCR) + 2 CAS/HUB + 2 ISA + 2 Edge)
TOTAL: 92 Exchange and server that Exchange relies on (Email only)
Read the rest of this entry »

I was taking a look at the Exchange 2007 blog and viewing some of the videos.

One videos called Exchange 14 Video shows some of the new exchange 14 management features. The other video I watched was called Microsoft Transporter Suite for Lotus Domino demo I was pretty surprised to see that one of the main parts of Exchange administration uses “power shell” to perform tasks. It’s amazing how that can be one of the main features of the way Exchange is managed.

One example showed migrating a mailbox and messages.
One of the engineers giving the demonstration was touting how images do not automatically showed up messages and said “in Domino that wouldn’t happen automatically, images show up automatically in Notes”

I wonder if he realized that this is not the default, and that this is a configurable option.

We have two servers in a cluster in Brazil. The hardware is HP ProLiant DL380 G5. This disks are 5 SAS 10,000RPM 146GB hot pluggable drives in a Raid5 Array. We have HP Insight Management Agents and HP Systems Management Home Page setup on those servers with the HP Event Notifier configured to send our admin group notifications for any hardware events though the SMTP server on the Domino server of the primary server in the cluster.

Today, the local administrator in Brazil sent us a photo of the front of the server. In the photo one of the drives had a red light on it. I connected to the server and looked a HP SMH and HP Integrated Management Log (IML). Sure enough, one of the drives was dead and had been acting up since January 31st. But why not notification?? Read the rest of this entry »

I have forever used Edit\Unread Marks\Mark Selected Unread when going through my email.

Today, I tried this on my 8.5 client, and noticed that there were the word “Insert” next to both Mark selected read and Mark selected Unread. Awesome shortcut!

What’s more is that this works in 7.03 as well, and I’m guessing probably every Notes release for a long time. The 7.03 client does not have the “Insert” helper next to the menu items though.

For months, I thought something was wrong with my mailbox, or at least cussed Lotus Notes search algorithm.

When I would search for something that I knew would produce lots of results, like Domino, I would only get 5 results. One day, I was going through and deleting some old messages, and I realized that if I deleted one of the 5 result messages, I could perform another search and another 5 would appear. Why was the 6th one hiding during the first search, and why am I only able to search 5 messages at a time. Frustrating.

Then, I clicked on the “more” twistie to expose the advanced search options, and noticed the “Max Results” button. This button only shows if your mailbox is full text indexed. I found that this was set to 5!!! I changed it to the maximum of 5000 and my mailbox searching life has been transformed. Everything works as planned now. I’m not sure how this got set to 5. I’ve even double checked our mail template didn’t have that as a default. No idea.

I manage an environment with over 1800 mailboxes that are greater than 2GB (including hub and cluster mate copies) and 66 above 10GB. The physical limit of a Domino database is 64GB.

However, anything beyond 2GB becomes very difficult to manage and here’s why.
1. Databases become corrupt very often.
2. Running Fixup or Compact on a weekly basis as part of Domino routine maintenance becomes problematic because the server is always running the maintenance on the databases. In fact, you’ll need to split up fixup and compact to run on specific directories during different evenings of the weeks. You may even want to run Updall on different directories different nights of the week.
3. Backups take forever and may not finish before the beginning of your work day.
4. Opening the mailboxes takes a very long time because it has to index the inbox every time you open it (if the user does not file messages to folders, which most don’t).
5. You pretty much can’t have full text indexing turned on for the server copies of the database because the full text indexing will be continuous around the clock on your server, not to mention more disk space required on server.
6. We’ve had some very strange things happen with Blackberry Enterprise Server accessing very large mailboxes. Specifically, We have had instances where a Domino server running Blackberry Enterprise Server (BES) can have frequent outages in IDTable code when working with large ID tables, which causes the server to crash. See technote titled “BES server crash in IDTable code”

7. If users have local replicas of their mailboxes, sometimes their hard drive is not large enough, sometimes the local replica becomes corrupt on the hard drive because no maintenance (fixup/compact) is ever run on it.
8. Creating new replicas onto new servers can take a week across the wire.

The list goes on.

These are real world examples of issues that I come up against every week. The best practice is to have quota policies in place that the upper management supports.

The second major thing to note here is that a proper archiving solution needs to be implemented. The local archiving that is built into Notes is often times not sufficient if you have so many enourmous mail files.

I posted last week about the SuperAdmin OpenNTF project. I installed it last Friday and let the agent run a while, and came in this morning to check out what it could report.

Keith Brooks commented in the last post that the ability to see the templates and ODS has been available in the Administration client since 6.5, plus the ability to modify the Administration Server (not see it). This is all true, but I still find the tool very useful for cleaning up ACL and administration server inconsistencies.

Here’s my review: Read the rest of this entry »

Yesterday, I posted about setting the administration server on all of my mailboxes at the suggestion of my colleague Jean-Yves Riverin

Well, today, I received yet another great tip from him. He pointed me to an OpenNTF project called SuperAdmin by project chef Declan Lynch

Here’s the synopsis:

Have you ever wondered if you have dead mailfiles sitting on your server taking up space, or if the all your databases are on the latest ODS version, or if the administration server in the ACL is set correctly on all databases, if so then this is the app for you.

Written by a notes admin for notes admins all the information that you need to make sure your Domino environment is running like a well oiled machine will be waiting for you once you deploy SuperAdmin.

I’ll setup it up and review it in a subsequent post.