Domino @ Symetrik Design

Today, while working in Argentina, I started receiving hundreds of notifications (events4.nsf) that database after database was corrupt. This was a bit worrying to say the least especially because several system databases were among the list of databases being reported.

This presents several problems.
1. Database corrupt (obviously)
2. Hundreds if not thousands of notifications emails clogging up our admin mailboxes
3. The main support team in HK are sleeping right now.
4. I have no direct way to connect to the server in India from Buenos Aires through PCAnywhere or Remote Desktop.

I was able to get to the server console via passthru through one of our hub servers in Hong Kong.

All of our sites are not directly connected to each other, but they are all connected to our main site in HK, so passthru actually comes in handy sometimes when working at another site.

When I connected to remote console, the first message I saw was “unable to record to log - log.nsf corrupt”

Now it’s really getting bad!

I have seen a backup agent that was not properly configured create similar problems where the backup agent is not utilizing an open file agent, or a domino specific agent. It grabs the files while Domino is still up and thus causes corruption.

It could also be anti-virus solutions.

The only thing I could think to do was to issue the “restart server” at the console.

This will automatically shut down the domino tasks and restart Domino 10 seconds later.

I figure that this would stop the mass flood of emails coming in, and if the server came back up it would automatically run a consistency check of the databases that were corrupt.

It would probably minimize the number of databases that were corrupted by just shutting the server down anyways. Hopefully, whatever was causing the corruption would go about it’s business after Domino got out of the way.

The joys of remote admining.

We are about to implement a webmail server in a remote site. This site has a newly installed mail server cluster. One on the LAN and one in the DMZ to serve webmail/Internet replication.

I had previously setup the Web SSO fields in the server document for the machine that is sitting out in the DMZ. I added the server name to the Web SSO for LptaToken document. We already have this in place in a couple of locations, so we’ve proved the concept and know it works.

I placed the domlog.nsf database that I wanted on the server.

Once all that was in place, I tried accessing the server’s URL successfully directly and logged into a test mailbox without problem.
Now time to test the Web SSO.

We have a DNS entry for notes.domain.com, this has a webmail redirection setup so that the browser is re-directed to the homemail server that the logging in user has a mailbox on. This is automatically done by a lookup on the person document.

Since we haven’t implemented our 2 clustered mail servers in many of our sites yet, many of the mailboxes are all on our hub servers in Hong Kong. We play some DNS tricks to make all this work and get the webbrowser to be redirected to the user’s home mail server, however, the home mail servers is just a C-NAME alias of the Hub1 server’s DNS entry.

So today, the web SSO wasn’t working properly.

I had this problem once before where is seemed like it intermittently worked. In reality, it worked but about 10 seconds after I logged in and was redirected, and then told that the login failed on the second server. If I clicked refresh on the browser, it went ahead and logged into the webmail account.

I realized that the token wasn’t yet valid on the second server. It was “too soon” in other words.

I used the DEBUG_SSO_TRACE_LEVEL=1 Notes.ini parameter to see what was going on during web re-direction and authentication. You can also use DEBUG_SSO_TRACE_LEVEL=2 to get even more information.

I was able to see the “ERROR: Token is expired” and see the time of token creation and token expiration as compared the time on the server.

Without the details in those extra logging levels, I wouldn’t have been able to figure out what the problem was.

We have our Windows 2003 servers set to synch their time with an Internet time source (NTP server). I double checked and made sure that I have previously configured the new server to synch it’s time with the same NTP server that the notes.domain.com server was using.

In fact it was configured and working properly.

The only thing I could do was to manually set the 2nd (or re-directed) server’s time 10 seconds ahead of the notes.domain.com server. This way when the token is created on the 1st server, then redirected to the second server, the token will have been valid for at last 8 or 9 seconds already depending on long the redirection took.

I’m not sure why the time wasn’t exactly the same on the 2 servers, because they were both syncing properly with the NTP server.

I’m guessing that the time might not be synching if it is only a few seconds off. Perhaps it needs to be more than 1 minute to be corrected.

Who knows. It’s working now though. Let’s hope it stays that way.

On all of my Domino servers, I load some of the Windows NT, 2000, or 2003 Resource Kit Utilities. See Microsoft KB article 927229 for the Windows 2000 Resource Kit Tools download page (You can download them individually and they’re free).

Here is the link to the Windows Server 2003 Resource Kit Tools

The individual tools I use are:
Kill.exe
now.exe
Sclist.exe
sleep.exe
srvinfo.exe
Timeout.exe
TLIST.EXE
WNTIPCFG.EXE (does not work on Windows 2003 unfortunately)

I put them into the c:\program files\rktools\ directory, which is the default if you install the whole package. I also put that directory into the path so that you can execute any of the commands from any directory or any script/batch file.

If you have a hung Domino process that refuses to shut down, go to the command prompt and type:
C:\>tlist
0 System Process
4 System
416 smss.exe
464 csrss.exe
488 winlogon.exe
536 services.exe
548 lsass.exe
940 svchost.exe
1128 spoolsv.exe
1152 msdtc.exe
1252 AWHOST32.EXE
1284 EmcPowSrv.exe
1296 RMServer.exe
1312 HbaHsMgr.exe
1388 svchost.exe
1428 ibmhpasv.exe
1444 IBMSPSVC.EXE
1464 nservice.exe
1476 IBMSPREM.EXE
1484 IBMSPREM.EXE
1600 NaviAgent.exe
1636 nsrexecd.exe
1700 nsrpm.exe
1712 nserver.exe Notes1/Domain: Lotus Domino Server
1728 NTRtScan.exe
1824 svchost.exe
1836 savapi2s.exe
1860 RaidServ.exe
2116 TmListen.exe
2168 nevent.exe
2612 nupdate.exe
2720 nreplica.exe
2728 nrouter.exe
2740 namgr.exe
2748 nadminp.exe
2756 ncalconn.exe
2764 nsched.exe
2772 nhttp.exe
2800 patrolat.exe
2844 ncollect.exe
2868 nrunjava.exe
2876 ntm_grab.exe OleMainThreadWndName
2884 nSMDemf.exe
2912 nSMDreal.exe
2920 nSMDsch.exe
2928 nSMDmon.exe
2936 nsmtp.exe
3044 svchost.exe
3312 namgr.exe
3340 alg.exe
3684 CNTAoSMgr.exe
3700 AD28BF.EXE
308 ncldbdir.exe
1744 nclrepl.exe
[SNIP]

I’ve cut the list short to save space.

Let’s say for example you want to stop the router task and from the Domino console “tell router quit” has no effect.

Notice in the list above, that the router is listed as “2728 nrouter.exe”

If you have hundreds of tasks, you can type:
C:\>tlist nrouter.exe
2728 nrouter.exe
CWD: d:\Lotus\Domino\
CmdLine: d:\Lotus\Domino\nRouter.EXE
VirtualSize: 1056780 KB PeakVirtualSize: 1057804 KB
WorkingSetSize: 14460 KB PeakWorkingSetSize:598896 KB
NumberOfThreads: 12
2732 Win32StartAddr:0×00434540 LastErr:0×00000000 State:Waiting
2292 Win32StartAddr:0×601158c0 LastErr:0×00000000 State:Waiting
1072 Win32StartAddr:0×601158c0 LastErr:0×00000000 State:Waiting
720 Win32StartAddr:0×601158c0 LastErr:0×00000000 State:Waiting
4616 Win32StartAddr:0×601158c0 LastErr:0×00000000 State:Waiting
5108 Win32StartAddr:0×601158c0 LastErr:0×00000000 State:Waiting
1572 Win32StartAddr:0×601158c0 LastErr:0×00000000 State:Waiting
4004 Win32StartAddr:0×601158c0 LastErr:0×00000000 State:Waiting
5664 Win32StartAddr:0×601158c0 LastErr:0×00000000 State:Waiting
2944 Win32StartAddr:0×601158c0 LastErr:0×00000000 State:Waiting
4344 Win32StartAddr:0×601158c0 LastErr:0×00000000 State:Waiting
2320 Win32StartAddr:0×601158c0 LastErr:0×00000000 State:Waiting
6.5.40.5086 shp 0×00400000 nRouter.EXE
5.2.3790.3959 shp 0×7C800000 ntdll.dll
5.2.3790.4062 shp 0×77E40000 kernel32.dll
6.5.43.6009 shp 0×60000000 nnotes.dll
6.5.40.5086 shp 0×621B0000 nxmlpar.dll
6.5.40.5086 shp 0×62320000 nxmlcommon.dll
7.0.3790.3959 shp 0×77BA0000 MSVCRT.dll
6.5.40.5086 shp 0×62150000 js32.dll
6.5.40.5086 shp 0×62350000 NLSCCSTR.DLL
5.2.3790.3959 shp 0×77F50000 ADVAPI32.dll
5.2.3790.4115 shp 0×77C50000 RPCRT4.dll
5.2.3790.3959 shp 0×76F50000 Secur32.dll
6.0.3790.3959 shp 0×766D0000 SHFOLDER.dll
5.2.3790.4098 shp 0×77D00000 OLEAUT32.dll
5.2.3790.4033 shp 0×77380000 USER32.dll
5.2.3790.4033 shp 0×77C00000 GDI32.dll
5.2.3790.3959 shp 0×77670000 ole32.dll
6.0.3790.3959 shp 0×762B0000 comdlg32.dll
6.0.3790.3959 shp 0×77DA0000 SHLWAPI.dll
5.82.3790.3959 shp 0×77530000 COMCTL32.dll
6.0.3790.4184 shp 0×7C8D0000 SHELL32.dll
5.2.3790.0 shp 0×71BB0000 WSOCK32.dll
5.2.3790.3959 shp 0×71C00000 WS2_32.dll
5.2.3790.3959 shp 0×71BF0000 WS2HELP.dll
6.5.43.6009 shp 0×62950000 ndgts.dll
5.82.3790.3959 shp 0×77420000 comctl32.dll
6.5.40.5086 shp 0×624D0000 NSTRINGS.DLL
10.2.2.1980 shp 0×10000000 ntk_hook.DLL
3.6.0.0 shp 0×4A800000 icuucgrp36.dll
3.6.0.0 shp 0×4AD00000 icudtgrp36.dll
6.5.40.5086 shp 0×625B0000 namhook.DLL
3.0.0.1280 shp 0×038D0000 nSMDext.DLL
10.2.2.1980 shp 0×03900000 nte_hook.DLL
6.5.40.5086 shp 0×62610000 nNTCP.DLL
5.2.3790.3959 shp 0×71B20000 mswsock.dll
5.2.3790.3959 shp 0×76ED0000 DNSAPI.dll
5.2.3790.3959 shp 0×76F70000 winrnr.dll
5.2.3790.3959 shp 0×76F10000 WLDAP32.dll
5.2.3790.3959 shp 0×76F80000 rasadhlp.dll
6.5.40.5086 shp 0×625D0000 nTCP.DLL
5.2.3790.3959 shp 0×5F270000 hnetcfg.dll
5.2.3790.3959 shp 0×71AE0000 wshtcpip.dll
6.5.40.5086 shp 0×044C0000 nFTGTR40.DLL
6.5.40.5086 shp 0×044E0000 gtr40nts.dll
6.5.40.5086 shp 0×04570000 nlxlid102.dll
6.5.40.5086 shp 0×04590000 nlxrt22.dll
6.5.40.5086 shp 0×0FD40000 nlxsum22.dll
6.5.40.5086 shp 0×0FDA0000 kvfilter.dll

The results show each nrouter.exe task and it’s Process ID (PID).

Once you know the PID, you can use the kill.exe command

C:\>kill 2728
This will show you that a process has been killed.
You should notice that the process disappears from the process list in the Windows Task Manager and from the Domino console list.

This works 99% of the time.

You can also type:
C:\>Kill nrouter.exe
This does work, but I have found mixed results. It works less often that specifying the PID.

You can also try to use the Windows Task Manager, select the Processes view, and click “End Process” on the hung process, but I have found that this does not work very often.

All of that being said, you should only kill processes like this if it is an extraneous situation and they will not quit via the Domino console. If you are successful in killing the process, you should try and reboot the server as soon as possible to have a “clean” Domino process and memory footprint. It would be assumed that if you are resorting to killing a process like this, the server cannot be recycled until the end of the business day. So typically if you need to kill a compact process, for example, because it is hung and eating up resources, but you cannot reboot the server because it is in the middle of a business day on a busy server - make sure and reboot it at the end of the business day to clean everything up.

I was recently asked about Hot-Keys in R8.

On the default Home page, it blatantly tells you what the key is to bring up a list of hot-keys.
Ctrl+Shift+L

Also, check out this weblink:
http://allhotkeys.com/lotus_notes_hotkeys.html

Today while trying to load my R8 client, my client got hung.

I have been having a strange problem where when I start Notes8, it sets there trying to load what I previously had loaded, and then offers up the dialog box “Do you want to replicate data with the server?” If I say either Yes or No, Notes stops responding.

I’m not sure why this is happening, but I’ve noticed some flakiness between switching between my admin ID and my personal ID frequently, which is how I work.

NOTE: I’m already on 8.01.

So I killed the task, which is nice. In older versions of Notes this just wasn’t possible. Notes was pretty hard to kill.

I decided to go to the command line and delete cache.ndk. I couldn’t delete the file because it was in use by another process. I thought this was strange, but it would explain why when this had happened to me before, and I had simply killed the tasks, restarted Notes and got the same result. A bit annoying really, and not much in favor of Notes 8 or 8.01.

So, since I couldn’t delete the cache.ndk, I switched the program directory and issued “NSD -kill” which kills all Notes tasks and generates an NSD. I was then able to delete cache.ndk from the data directory and restart Notes.

I waited a few seconds before clicking on the “yes or no” box when it asked if I wanted to replicate data to give Notes a little time to catch up and this time it loaded fine.

I’m not sure if it was deleting the cache or if it was having a little patience. Either way, I’m in there, but the real story is that Notes has come a long way and being able to kill the tasks, and/or run NSD -Kill and be back into Notes without restarting Windows is a god send.

I feel like talking like a cave man because the concept is like starting fire when people start to understand the benefits.

For a long time, I’ve been a pundit of “DON’T USE STATIC IP CONNECTION DOCUMENTS!” Not everyone will listen, maybe because I sound like a radical cavalier consultant admin who knows everything - offering up such new fangled ideas to an organization who does not have a global active directory container or single company-wide DNS suffix.

Here’s why STATIC is so bad:

We have a situation in a large office where which will be changing it’s IP address range to fit into the global IP address scheme standards. This means that all of the servers have to have their IP address changed. Excellent!…finally following up with some standards.

The question came up on how this will affect Lotus Notes clients. The first thing I suggested was to go to a few of the clients to see how they are configured because nobody has a clue. I’m guessing that they all have static connection documents, which are not a 100% bad thing as long as the IP address field of the connection document has a resolvable name for the server. For example:the IP address field should be Notesserver which could be resolvable either via WINS, or by DNS if the server’s host name has a DNS entry and there is an alias (C-Name record) for just the hostname.

This is another great example of why Domino server application name should always be the same name as the computer host name. It makes life so much easier, and it’s so simple to do. It’s not skin off of anyone’s back. If your Windows 2003 server name is dominoserver, and your Domino server name is Dominoserver, and you run WINS, then dominoserver will automatically be added to WINS without any further interaction.

If your hostname is the same as the Domino name, you run WINS, and all of the Windows/Notes Client machines are in the same active directory (thus utilizing the same WINS infrastructure) then you don’t even need a connection document.

If you want to be “sure” then create a connection document to dominoserver with the name “dominoserver” in the IP address field.

If you use DNS instead of WINS, the principle is the same. Create your DNS entry for your server dominoserver.domain.com. If you can, create an alias for that DNS entry as just the hostname, which is dominoserver. This means that you’ll resolve both of these on a windows client that has the DNS servers for your organization defined (the windows clients can receive DNS server IP addresses automatically from DHCP).

c:\>nslookup dominoserver.domain.com
Server: dns1.domain.com
Address: 10.0.0.10

Name: dominoserver.domain.com
Address: 10.0.1.200

c:\>nslookup dominoserver
Server: dns1.domain.com
Address: 10.0.0.10

Name: dominoserver
Address: 10.0.1.200

This means that both names can be resolved without problem. This also means that you do not need a connection document. If you want to be “sure” - you can create a connection document for dominoserver and in the IP address field, put dominoserver.domain.com.

Some organizations will rely on the “Append primary and connection specific DNS suffixes” or “Append parent suffixes of the primary DNS suffix” or “Append these DNS suffixes” settings in the TCP/IP settings on all the clients machines. You will see this in organizations that may not use WINS, or have DNS setup in such a way that they cannot create an alias for the dominoserver.domain.com. The latter is usually the case in organizations that have multiple DNS suffixes under one root and are unable to create host name aliases for anything under the subdomains.

What happens is that there will be a DNS entry for dominoserver.domain.com, but no alias for dominoserver. So typing nslookup dominoserver produces no results. However, on client machines that have the “append…..” setting, the client will automatically append, the .domain.com onto the NSLOOKUP request, which then in turn finds the DNS entry.

This can create problems on servers or clients that do not have the “Append….” settting defined or turned on.

With all of the above said, you can see how much of a benefit it can be to have all of your clients configured with no connection documents, or with connection documents that have a resolvable name in the IP address field.

If your server’s IP changes, because of some outside force that requires that it must be changed, you can rely on WINS or DNS resolution on each of the client machines and you do not have to worry about any client downtime.

If you have configured each of the Notes clients with connection documents and those connection documents point to an IP address, then you must change each and every connection document. If the office is 20 staff, it’s not a big problem. What if it’s 400? That’s alot of work, alot of unnecessary work.

Another very helpful thing is to put the IP address of the server in the Net Address Field of the Notes Network Ports tab in the server document.

The benefit here is that when one server tries to connect to another server, this field is used. For example, when server A does not have a connection document to Server B, but they are in the same Notes Named Network (NNN), mail will route from Server A to Server B and vice versa. This is because if Server A cannot resolve the Server B’s name (or perhaps the Domino name is not the same as the hostname), Server A will know the IP address of Server B based on this field and be able to connect to it.

This is also used when servers are in a cluster. When a client connects to his home server, a cluster cache is stored in the clients machine. This cluster cache stores the IP addresses of the servers in that cluster. IF the IP address is stored in the Net Address field instead of a name, then the client will know and connect to the second cluster machine on the IP address. If a name is stored, then the name must be resolvable by the client machine or the client machine cannot connect to the second cluster server in the event that the first cluster machine goes down.

Today,

I received an email/phone call/IM about a super-VIP user who’s mailbox in South America had a problem.

This user travels frequently, has a 23GB mailbox, and refuses to carry a laptop. He travels from his Swiss office to Hong Kong, Singapore, and South America.

When he arrives in Argentina, he expects his replica to be up-to-date without problems. The issue is that the database is 23GB and the comm links to Argentina are very slow, so sometimes, replication delay can be up to 2 or 4 hours.

What I suggested that we do is to create his mailbox on the clustered servers in Argentina and then get the Swiss and Argetina local IT staff to coordinate with us every time he is going to travel so that we can change his home mail server. This will mean that mail will not need to replicate to the Argentina server from Switzerland, through the hubs in Hong Kong before he will receive it in his Inbox.

For outbound mail, there is not an issue. Just set the home mail server in the location document on the desktop machine in Argentina to be that of the primary mail server in Argentina. The Domino server could care less what the person document says as long as it’s outgoing mail.

A couple of weeks ago, I was in Argentina on some other issues and got involved in this case since this is a super-VIP user, and since the local IT support staff do not have rights to create databases or replicas on the Domino servers.

The local IT support staff had created a local replica on the desktop machine that this user uses when in Argentina from the Hub in Hong Kong. It took 5 days to replicate and something didn’t seem quite right with the database.

I fan fixup, and then compact -C on the desktop machine manually. Then I created a replica on the secondary clustered mail server in Argentina. I did this on the secondary server so that I could run some maintenance on it and know that the secondary server wasn’t that busy. I created the new replica, ran fixup, compact -C, and Updall -R. Then I watched as replication started from the Hubs to the secondary Argentina server.

Once, I was satisfied that the replicas weren’t corrupt, I created a third replica on the primary Argentina mail server.

I left it as it was and let replication from the hubs catch up with the Argentina servers.

Over the weekend, we got a notification that the super-VIP user would be showing up in Argentina and the local IT staff had the diligence to check on the database to make sure that it looked right.

To their dismay, the newest message in the Inbox was from March 5th (20 days ago). They frantically notified us, and I began to take a look.

I found this message on our Hub servers:
Error encountered replicating folders from mail\mailboxname.nsf to NotesAR2/DominoDomain mail\mailboxname.nsf: Folder is larger than supported, cannot perform operation.
Unable to replicate NotesAR2/DominoDomain mail\mailboxname.nsf: Folder is larger than supported, cannot perform operation.

After running fixup, compact -C, and updall -R which had no effect, but we did notice that replication was actually caught up because all of the recent documents that should be there were in the All Documents view. The Inbox view was not displaying them properly though.

I began to search through the Lotus discussion forums and came across this technote which described the situation:
http://www-1.ibm.com/support/docview.wss?uid=swg21102463

I used NotesPeek to look at the $Inbox properties and sure enough, it had reached the 50 limit.

I changed the “Remove documents not modified in the last” from 90 to 15 days on the problematic replicas of the database and then watched as replication caught up.

(exact setting location)
Select the local replica database icon -> Choose File -> Replication -> Settings, and on the “Space Savers” tab, change the numeric value next to “Remove documents not modified in the last” from the default of 90 to 15 days, and then click OK.

After 80 minutes of replication, I opened up the database and all was fine!

The directory catalog is supposed to be used for mobile users so that they can address mail to users or groups remotely without having to have a replica copy of the huge NAB on their laptop. For some organizations, this is not a big deal, they can just create a local replica of the NAB if it’s small enough.

For other organizations the NAB is 1GB plus in size, and managing that on a local replica for all users, and even the possibility of security vulnerabilities can be a big issue.

Other environment use dc.nsf so that their users can search via first name instead of last name. (seriously, it happens).

Let me explain how it works:
Set it up on 1 server only. This is done in the server document, under the server tasks tab, under the directory catalog sub-tab.
In the dc.nsf, configure it so that aggregation only runs on that server. This is set by opening dc.nsf on the server you want it to run, select the configuration view, open the configuration document and supply the server name in the ” Restrict aggregation to this server:” field.
Once the dc.nsf is created, you can manually create replicas on the servers you want to exist with the admin client.

If the dc.nsf becomes corrupt or out-of-date for some reason, run these commands to manually update it on the server where the task runs.
To manually update the directory catalog, use these commands:

tell dircat quit
load dircat dc.nsf -r (rebuilds the dc.nsf then shuts down)
load dircat (required because the previous command shuts down the task when finished).

Thought this is an option in the notes client when sending email, you can easily get around it by clearing the $KeepPrivate field.

Or alternatively, just his the Printscreen key on the keyboard or use any other screen capture software.

Last night, one of our admins accidentally replicated the R8 pilot server address book to a R6.5.6 production server address book through his notes client.

This royally screwed up the 6.5.6 address book. Some of the servers rebooted themselves, mail wouldn’t route, dc.nsf appeared to be out of order, etc.

We fixed it by manually replacing the address books on the bad servers at the file system level with a good copy from another server that hadn’t replicated the bad design yet. Another option to keep open is to keep a local replica of the NAB on all of your admin clients, and set your admin clients only to replicate with the server ever hour, 2 hours, or 4 hours. This way, the local replica copy of the NAB on the admin clients is pretty much up to date, but if something happens, it gives you a bit a time to discover that something is wrong. It’s not a perfect solution, but good (possible) option to have in place when you notice that something is very very wrong. A scenario might be that you discover your NAB is bad, and your local admin client hadn’t replicated for 3 hours and does not have the bad design. This can save you.

Of course, there is always the backup from the night before, but then you’ve lost all of your changes from the morning or earlier in the day. In busy environments, this can be alot of work.

When we came in this morning, some users were complaining that the dc.nsf wasn’t accessible. Upon further investigation, what happened was the when the admin replaced the NAB at the file system level, they didn’t delete the names.ft (full text index directory files) and re-create the full text index of the NAB. This somehow interfered with the dircat process which runs on one of the servers and updates dc.nsf every 4 hours.

When we searched some of the NAB for a person’s name, the results would produce 10 or 15 names and not the name we searched for, and other oddities.

After we manually deleted the full text indexes and re-created them, things went back to normal.