Symetrik Domino Consulting

Yesterday was a rough day being a Domino mail server admin.

First, let me explain our environment. Most of our company is on Domino and Notes 6.5.1.-6 (mostly Notes 6.5.4).

There is one division that refuses to move to Notes because they love outlook, and their main excuse is that there is the 100 mail rule limitation of Notes. They actually use email as a system like I’ve never seen. They manage a fleet of ships, and each ships sends different kinds of messages, these messages are sent to groups that are nested several times, so many many people receive them. Some people need them for one reason or another. These messages also go into a linux application server and are search able via a web interface.

So basically, they cannot live without having sometimes 200 mail rules. They are pushing electronic mail to the boundaries. I’m sure there is a better way for them to do all of this, but who has the time to pitch a better idea to them, especially when they have no budget.

So, these guys access Domino mail files on 3 servers that we have that serve as both passthru server to everyone in the company, and SMTP/POP3 for this shipping divison. They access the server either with outlook, outlook express, or exchange client via POP3. A few of them are using the DAMO or Outlook connector. It’s a real mish-mash of clients out there. A huge mess and nightmare to support.

I don’t know if you have ever seen POP3 on a Domino server, but I wouldn’t recommend it. The servers are literally so busy, you can hardly issue commands on them sometimes.

We require authentication for SMTP connection so that these guys can send outgoing (from their client) mail.

What happens occassionally is that one or two of their PCs (or more) will get a virus, and start SPAMMING to no end. Since, they are authenticating to these 3 (passthru, SMTP, POP3) Domino servers, they are allowed to route mail through them. The virus, literally serializes the FROM address, so we don’t exactly know which user it is, and have no way of doing anything about it.

The main problem is that it tried to send Internet mail out, and alot of the messages are either blocked, or simply invalid domains. We have an outbound SMTP hub gateway, and initially these shipping users with the viruses were clogging up the outbound SMTP hub gateways, and legitimate Internet mail could not be delivered. So we allowed these 3 servers to deliver mail directly to the Internet, so that only the shipping mail would be affected (we are in a battle to force them to move to Notes client for glaringly obvious reasons).

What started happening in the last couple of days is that there is a particular clever SPAM virus, or malicious person who has control of one of their machines. We receive hundreds of SMTP connections to one of these servers, this in turn takes up so much server resources, that the router task cannot deliver legitimate mail to local Domino mailboxes, or mailboxes on servers sitting in the same Notes Named Network (NNN). Additionally, many of these messages are to invalid Internet domains, so what resources the servers does have left after answering the SMTP connections is to try and connect to invalid or expired Internet domains to deliver these SPAM messages.

We found that there wasn’t much we could do about it. We looked at the from address which looked something like “agds” jgo@ourdomain.com, “bgssd” jgo@ourdomain.com, “zsd;js” jgo@ourdomain.com

This makes it difficult to go through the mail.boxes (we have 4 on each server) and sort by sender and delete all of these rogue messages.

The only thing we can really do is go to the inbound SMTP controls and refuse messages from jgo@ourdomain.com. The issue is that a few hours later, the FROM address will be serialized and it will turn out to be a different FROM address.

Any ideas?

On top of that, our outbound SMTP gateway started coming up with lots of these types of messages:

Router: [0000000C] Skipping message because ReadyToSweep

We had never seen them before, so I looked it up and found this IBM technote:

http://www-1.ibm.com/support/docview.wss?uid=swg21260649

It turns out that our PR/Marketing department had sent out our 2007 annual results. They used a group which contains everyone in the company, and they also have their own personal address book entries with shareholders, business partners, etc. etc.

It turns out that the servers were overloaded trying to send to invalid or expired addresses.

Both of these things were happening at the same time, and I was beginning to wonder if the roof was going to cave-in down us next.