Today, the local administrator in Brazil sent us a photo of the front of the server. In the photo one of the drives had a red light on it. I connected to the server and looked a HP SMH and HP Integrated Management Log (IML). Sure enough, one of the drives was dead and had been acting up since January 31st. But why not notification??

I took a deep look at NotesBR1 the primary server in the cluster to determine why we did not receive any notification that the drive had gone bad.

The event notifier is setup correctly, and I was able to send two Test Trap Messages which were properly received.

The HP Integrated Management Log (IML) shows this:

We should be receiving email notifications for each of these warnings, especially the critical ones.

I noticed this in the Event Log:

This is from the Domino server log:
02/07/2009 07:11:59 AM SMTP Server: Starting…
02/07/2009 07:12:03 AM SMTP Server: Started

When the Event Notifier (CimNotify) service starts up, it tries to open a connection to the SMTP server to make sure it is configured properly, but does not send a message. If it cannot open an SMTP connection, the service fails and generates this event log event.

What is happening is that the hardware events (errors/warnings) are generated when the server starts. Notice the time 7:10AM, 7:11AM, and 7:12AM. This is right after the weekly Saturday morning 7AM maintenance reboot. So basically, the Event Notifier doesn’t start because Domino has not finished starting up yet so it cannot open an SMTP connection, and then the hardware events are generated and appear in IML and cannot be sent via email because the Event Notifier service is not running.

Domino eventually starts up and Event Notifier eventually starts – so that’s why I was able to send test trap messages.

So the bottom line is that there is a flaw in the way that I’ve configured the Event Notifier on several of the servers. This would explain why over the past several months, we only receive Event Notifications from NotesBR2 and other even numbered (secondary) cluster servers. The even numbered servers are pointing to odd numbered (primary) servers as their SMTP servers. While NotesBR1 (and other odd numbered servers) are pointing to themselves as SMTP servers.

The problem does not occur on the even numbered servers because when the Event Notifier service tries to start, it connects to the odd numbered server’s SMTP server and the Event Notifier service proceeds to startup properly.

Happy Days!

p.s. Yes, the title is a reference to Get Smart.

I’ve been playing with the HP Systems Insight Management Console for the last few months. It’s a great tool and lots and lots of granular monitoring, patch and firmware deployment, etc.

I just realized this week that you could go to HP’s website and install what’s called a Proliant Support Pack.

make sure and set some settings in SNMP or the agents won’t work when you load up HP Systems Management Homepage.

You’ll need to go to the Service Panel and right click on SNMP Service. On the Traps Tab, you’ll need to specify a community of public (or whatever might be the standard at your organization).

Go to the security tab, and check the “send authentication trap”
For public rights should be “READ ONLY”
for private rights should be “READ CREATE”

In the “Accept SNMP Packets from these hosts” you will need to specify 127.0.0.1 and localhost.
If you have a HP Systems Insight Management Console setup, you’ll need to specify the hostname or the IP address of your HP Systems Insight Management Console server.

These settings may be set for you if you use the HP Smart Start DVD to install Windows 2003.

If you need to install SNMP after you install Windows, be careful as it automatically selects IIS server when you select SNMP when adding programs to get the SNMP service installed.

I also had a bit of trouble installing SNMP after Windows 2K3 Service Pack 2 had been installed. I had to insert a Windows 2003 Server CD, and also download and extract the SP2 install image to a directory because the SNMP installation required files from both install packages before it would install.

I tricked SP2 by clicking on the download image and letting it expand, and then copying the expanded folder to another location and then clicking cancel, which then deleted the original expanded SP2 install directory.

I have since found out that there is a command to expand it.
WindowsServer2003-KB914961-SP2-x86-ENU.exe -x:c:\temp\sp2\
to expand it into the c:\temp\sp2\ directory.

Now that you have SNMP and HP Systems Insight Management Home Page working…you can check to see if your hardware has any problems, what type of HP system you have and what and how many of each component you have.

If you have a HP Systems Insight Management Console, Version control agents (on the client server), and an up-to-date Version Control Database (on the Management Console server), you can determine what version of ROM and drivers you have compared with what has been downloaded from HP on your Insight Management Console and is available for install from your Insight Management Console server.

This is a great way to keep your server up-to-date.

Additionally, you can “push” the latest updates to a single or list of (client) servers from the HP Insight Management Console servers. It works very well, and its very simple once the whole thing is already setup. Sometimes discovery can be a little flaky and you need to specify your servers on the HP Systems Insight Management Console, but otherwise, it’s very good.

But what if you don’t have a spare server laying around to use as a HP Systems Insight Management Console, with the Version Control Database on it?

You can download the Prolian Support Pack as mentioned above for your server and apply all the latest baseline HP software and Windows drivers to a single server or even push that to a group of local servers.

It’s kind of the same concept without the dedicated HP Systems Insight Management Console server.

One thing to note is that after you download and install the Proliant Support Pack from HP, you’ll also need to download the ROM updates for the CPU, ILO and ILO2 Boards, and Array Controllers seperately from the Proliant Support Pack. Once you have downloaded the correct ones, you can put them in the same directory with the other Proliant Support Pack files and apply them all at once.

Very cool stuff.

One thing that I have to complain about though is that it is VERY VERY difficult to find individual downloads on the HP site. It’s almost impossible even using google and their own search engine.

Here’s a very useful URL. START HERE -> http://h20180.www2.hp.com/apps/Nav?h_pagetype=s-001&h_lang=en&h_cc=us&h_product=241435&h_client=S-A-R163-1&h_page=hpcom&lang=en&cc=us