Domino @ Symetrik Design

For a long time, I’ve been frustrated with Domino on the massive amount of messages that we sometimes receive over the weekend, late at night, or early in the morning indicating that a mail.box or some other database is corrupt.

Here’s a list of the common event generators/notifications we receive and a step by step guide on how I setup corrective actions to fix the problem and prevent the event task from spewing thousands of notification messages.

• Event/notification: Error compacting mail\somedatabase.nsf: Database is corrupt — Cannot allocate space
• Corrective Action: New event handler runs “load compact -c <database name>”




• Event/notification: Router: Mailbox file mail1.box is corrupt
• Corrective Action: New event runs “load fixup <database name>”




• Event/notification: Unable to update activity document in log database for bookmark.nsf: Database is corrupt — Cannot allocate space
• Corrective Action: New event handler runs “load fixup log.nsf”

This should significantly reduce the amount of messages that are generated by runaway event monitor notifications.

The following events cannot have event handlers run on them.

• Event/notification: Unable to store document in MailServer1/MyDomain mail\usermailbox.nsf (NoteID = 1264886) from mail\usermailbox.nsf (NoteID = 1209126): Database is corrupt — Cannot allocate space
• Reason: the command would have to be generated on remote server. No possible way to pass an argument to remote server on R6.5.6.




• Event/notification: Unable to replicate MailServer1/MyDomain mail\usermailbox.nsf: Database is corrupt — Cannot allocate space
• Reason: the command would have to be generated on remote server. No possible way to pass an argument to remote server on R6.5.6.




• Event/notification: Database is corrupt
• Reason: This event is generated by the router task based upon messages in the mail routing view in the log. The database that is corrupt is listed on a seperate line in the log than the “Database is corrupt” text, so there is no way to determine by <string> which database to run a command on.

Read the rest of this entry »

We use Message Labs for anti-spam prevention service. They are pretty good at stopping most of the spam, but not all.

For those of you who are not familiar, we point our MX records to Message Labs servers that they specify. Then Message Labs SMTP servers filter out spam and viruses using their own algorithms and forward on the good messages to our inbound SMTP gateway.

The problem is that Message Labs can’t block backscatter as they would block valid non delivery failures in the process.

How are you coping??

I’ve been investigating why our primary mail routing hub server is taking so long to route messages recently. The behavior started a few days ago (or maybe it’s been going on for a while and we didn’t notice it).

• Messages will queue in the mail.boxes for anywhere from 5 to 15 (maybe longer) minutes.
• They do NOT have the -check in progress- (which sometimes take place when Trend Micro ScanMail or IQSuite, our mail compliancy capturing software, is holding up the router).
• Then suddenly they all route at once.
• Then the process will start over again.

Read the rest of this entry »

I have just enabled Message Tracking on an R6.5.6 server.

All I had to do was to enable message tracking in the server’s configuration document.
I added LocalDomainServers to “Allowed to track messages” field, but left everything else as is.

Once that was enabled, I issued “load MTC” on the server in question.

I initially got these errors when the task loaded:

MT Collector: Initialization failed: File does not exist
MT Collector: Unable to use directory e:\Lotus\Domino\Data\mtdata: File does not exist

Read the rest of this entry »

Today at the office we had a situation that had never come up before. We recently created a new user, which has the same name as a very old user.

A particularly important group of users regularly send to the old user, using just his common name, example “guru”

Suddenly, these important users started getting the Ambiguous Name Dialog Box which I’m sure that they have never seen before, and I’m sure they hit enter, and the result is that the new user, who is listed first in the ambiguous name dialog box receives the email.

Important users complain, heads roll, management wants the new user’s name changed (even though his name follows our newly implemented naming convention). Read the rest of this entry »

We were getting a few strange sounding reports from a user who was trying to send email to 3 groups.

The first message was:
“A copy of this message was sent to 180 recipient(s), but the Internet format of this message was not sent to the following 34 recipient(s); <address@domain.com>, <address@domain.com> etc.”

The second message was:
“Document has invalid structure: mail.box”

We closed the client, deleted mail.box, opened the client, and let it re-create mail.box and all was fine.

We have sites in South America where network connectivity is very unreliable. We have MPLS lines connecting the remote offices in those locations, and we also have Internet access from each of the offices.

When setting up our replication/mail routing architecture in these locations where connectivity is unstable we got a little creative.

First, we have our hub servers in the Home office in Hong Kong. These hub servers do all of the replication from Hub to spoke. This cuts down on the overall replication overhead on the network. The idea is that cluster cache doesn’t have to be rebuilt every time if we have only the hubs replicating to the spokes. At the end of the day, network utilization and time required to replicate is actually lower.

We also put clusters in each site. This alleviates downtime in the event that one of the servers goes down. Since we have clusters, we can set the connection document to replicate with the cluster instead of the individual servers. This provides much benefit because replication still occurs if one server is down. It also reduces the network utilization in half because replication will only occur one time, to the cluster, instead of twice to each individual server. Read the rest of this entry »

We have two hub servers in our environment that are clustered.

We use these as replication hubs to all of our spokes (local and remote sites). Additionally, these act as sort of a mail hub since all of our remotes sites are not directly connected to our home office site. Lastly, they also serve as outbound SMTP gateways.

Any mail that is sent from Chile to a user in Spain gets routed through Hong Kong since their is not a direct connection from Chile to Spain.

All servers in each site are in their own NNN (Notes Named Network). The Hong Kong site servers are in their own NNN. The hubs are in their own NNN.

This allows us to route mail through Hong Kong where necessary using mail connection documents. Read the rest of this entry »

Yesterday was a rough day being a Domino mail server admin.

First, let me explain our environment. Most of our company is on Domino and Notes 6.5.1.-6 (mostly Notes 6.5.4).

There is one division that refuses to move to Notes because they love outlook, and their main excuse is that there is the 100 mail rule limitation of Notes. They actually use email as a system like I’ve never seen. They manage a fleet of ships, and each ships sends different kinds of messages, these messages are sent to groups that are nested several times, so many many people receive them. Some people need them for one reason or another. These messages also go into a linux application server and are search able via a web interface.

So basically, they cannot live without having sometimes 200 mail rules. They are pushing electronic mail to the boundaries. I’m sure there is a better way for them to do all of this, but who has the time to pitch a better idea to them, especially when they have no budget. Read the rest of this entry »