Sonntag, 8. November 2009

Monitoring an Apple Airport station using Nagios

Recently I was fed up with the problems with my Apple Airport stations (music interruptions because of network or other problems) and wanted to monitor them to finally see what's really happening in my Airport network.
So, I saw that the Airport stations support SNMP and thought it should be an easy task to monitor them using check_snmp. BIG failure ;-)
It was one of the most complicated things I have ever managed. But, I was successful.
So, to spare someone else two days of work, I decided to write down how it works.

Things you need:
1. The Apple Airport MIB AIRPORT-BASESTATION-3-MIB (from http://docs.info.apple.com/article.html?artnum=120227)
2. A working Nagios, either on your Mac or any Linux Server
3. A lot of patience ;-)

First of all, you have to integrate the MIB into the server where your Nagios runs, so that snmpget, snmpwalk and check_snmp can find it later. To do this, you should copy it into the /usr/share/snmp/mibs/ directory and name it AIRPORT-BASESTATION-3-MIB.txt.
Then you need to modify the .index file to include a line like:
AIRPORT-BASESTATION-3-MIB AIRPORT-BASESTATION-3-MIB.txt
This means, that the file you just copied contains the AIRPORT-BASESTATION-3-MIB and from now on you can reference it by that name.

Now you should test if you can already see the SNMP values of your Airport station using the snmpwalk command:

snmpwalk -v 2c -m AIRPORT-BASESTATION-3-MIB -c public 10.0.0.51 enterprises.apple.airport

Some notes on this:
1. 10.0.0.51 is the IP-Address of my Airport Express, substitute it with YOUR proper IP
2. The value following the -c switch is NOT the SNMP-Community-String, but the password of your APEX. (Although public is the default for both ;-))
3. enterprises.apple.airport is the root of the value tree that is walked by snmpwalk.

After you have executed this command, you now know which values are available to you to monitor.

Now you can execute two commands to further test your environment:
snmpget -v 2c -c public 10.0.0.2 AIRPORT-BASESTATION-3-MIB::sysConfFirmwareVersion.0
or
snmpget -v 2c -c public 10.0.0.2 -m ALL sysConfFirmwareVersion.0

If your Airport MIB was properly integrated, both commands should have the same result, in my case:
AIRPORT-BASESTATION-3-MIB::sysConfFirmwareVersion.0 = STRING: 7.4.2

The "-m ALL" switch simply means "lookup value in all available MIBs" and as we have integrated the Airport MIB in the mibs directory, it should work fine.

OK, so far so good. I got until this point fairly easy as well. But now comes the "Big Price Question": How do I get to the real interesting values like

wirelessStrength."00:1C:B3:AF:49:26" = INTEGER: -24

which were output by the snmpwalk command ? Because that is what I really want to monitor, I want to see if my music dies because the signals go down.
At first I thought, OK, easy, just carry on like before, I just need a command like:
snmpget -v 2c -c public 10.0.0.2 -m ALL wirelessStrength."00:1C:B3:AF:49:26"

BIG failure !!!

This just produces the following error message:
wirelessStrength.00:1C:B3:AF:49:26: Unknown Object Identifier (Index out of range: 00:1C:B3:AF:49:26 (wirelessPhysAddress))

OK, and now comes the wizardry, that took me the next two days to figure out, and you, my dear reader, just two minutes ;-)

So, what problems are we facing here ?
snmpget (and snmpwalk as well) cannot use the value wirelessStrength like they could use the value sysConfFirmwareVersion. I don't exactly know why, but I suppose it is because these values are part of a SEQUENCE in the MIB. (By the way, do read the MIB, it is quite interesting !)
So the first thing we have to do is to transform the value into its numerical OID.
This is done with the command:
snmptranslate -On AIRPORT-BASESTATION-3-MIB::wirelessStrength

which gives us

.1.3.6.1.4.1.63.501.3.2.2.1.6

OK, pretty obvious, eh ! :-)
Finished ? Oh no !
Now we need to retranslate this value into another textual representation using

snmptranslate .1.3.6.1.4.1.63.501.3.2.2.1.6

which gives us

SNMPv2-SMI::enterprises.63.501.3.2.2.1.6

And this is a value that we can already "snmpwalk", like this:
snmpwalk -v 2c -m AIRPORT-BASESTATION-3-MIB -c public 10.0.0.51 SNMPv2-SMI::enterprises.63.501.3.2.2.1.6

which gives an output in my case of

AIRPORT-BASESTATION-3-MIB::wirelessStrength."00:1C:B3:AF:49:26" = INTEGER: -24
AIRPORT-BASESTATION-3-MIB::wirelessStrength."00:25:00:3F:05:ED" = INTEGER: -52

OK, we've come quite far now: We can ask our Airport station for the Signal Strength's of its connected clients with a "simple" command.
But how do I get an individual value, like the Signal Strength of my Client with the wireless MAC of 00:1C:B3:AF:49:26 (you already knew these were wireless MAC's, didn't you ? :-)) ?
OK, I just tell you how it goes and then explain it.

snmpget -v2c -c public 10.0.0.51 SNMPv2-SMI::enterprises.63.501.3.2.2.1.6.\"00:1C:B3:AF:49:26\"

which gives us:

SNMPv2-SMI::enterprises.63.501.3.2.2.1.6.17.48.48.58.49.67.58.66.51.58.65.70.58.52.57.58.50.54 = INTEGER: -24

OK, so now I now that the Signal Strength of my client is "-24", which means the original signal is damped by -24 DBm.

If I now want to know what is going on in my network, I just have to compare this value, the Signal Strength, to the other relevant value, the Signal Noise (wirelessNoise in the MIB) and just have to take the difference of the two values to get the SNR (Signal to Noise Ratio), which should be at least 30 DBm.
(If all this sounds unfamiliar to you, read about it on Wikipedia)

Good, so now we know how to get all these values by hand, but how do we get them into Nagios ?

This is the second part of this article. Actually, at first I thought I should display the individual values using snmp_check, but that did not work at all, as snmp_check is too inflexible. Then I used snmpget directly, which worked, but eventually I found out that displaying values that way is also not the best decision, as I have to know the wireless MACs that I want to display beforehand, which I don't do. So the best would be to display multi-line values using snmpwalk. This actually worked. You just see one line in the overview display of Nagios, but if you click on the individual service you see all lines. I used that setup finally.

Now, how do my Nagios file look like ?
First, I modified the snmp.cfg file inside /etc/nagios-plugins/config on my Debian Linux based Nagios server.

I added the following lines:

define command {
command_name snmp_apexstrength
command_line snmpwalk -v2c -c public $ARG1$ SNMPv2-SMI::enterprises.63.501.3.2.2.1.6 -m ALL
}


define command {
command_name snmp_apexnoise
command_line snmpwalk -v2c -c public $ARG1$ SNMPv2-SMI::enterprises.63.501.3.2.2.1.7 -m ALL
}

These lines define two commands, snmp_apexstrength and snmp_apexnoise, with one parameter, the IP address of the Airport Express (or Extreme) you want to monitor.

The other file you need to edit is the host definition file for your APEX inside /etc/nagios3/conf.d
In my case, it looks like this:

tornado:/etc/nagios3/conf.d# cat host-apexkueche_nagios3.cfg
# a host definition for the gateway of the default route
define host {
host_name apexkueche
alias Airport Express Kueche
address 10.0.0.50
use generic-host
}

# Define a service to report the signal strength reported by the wireless client

define service{
use generic-service ; Name of service template to use
host_name apexkueche
service_description Signal Strength
check_command snmp_apexstrength!10.0.0.50
}

# Define a service to report the noise reported by the wireless client

define service{
use generic-service ; Name of service template to use
host_name apexkueche
service_description Signal Noise
check_command snmp_apexnoise!10.0.0.50
}

Here you can see how the previous commands are used.
Well, that's basically it :-)

One addition: If you want to use individual services, you can modify your snmp.cfg like this:

define command {
command_name snmp_get
command_line snmpget -v2c -c public $ARG1$ SNMPv2-SMI::enterprises.63.501.3.2.2.1.7.\"$ARG2$\"
; snmpget -v2c -c public 10.0.0.51 SNMPv2-SMI::enterprises.63.501.3.2.2.1.7.\"00:1C:B3:AF:49:26\"
}

Here you see two versions of the command: The commented out version is the original command, without variable substitution. The actual command uses two variables, ARG1 for the IP Address and ARG2 for the wireless MAC address.

If you would use this command, your Nagios host definition file would need a line like this:

check_command snmp_get!10.0.0.51!00:25:00:3F:05:ED

OK, this was what I did this weekend. I hope you enjoy it,

Bernhard