Monthly Archives: December 2014

You are here: Home . Archive

Nagios event handlers for NRPE

I used to use custom scripts to monitor services and get alerts of failures till Bas Moussa introduced me to Nagios. Monitoring and self-healing has been flowers and rainbows since.

Nagios event handlers allow you to put in place measures to self-heal your services. The Nagios documentation on setting them up is superb.
How do you do this for a service on a remote host that you monitor using NRPE? There’s this approach

Using it though, you’d have to send arguments to NRPE which, Nagios points out in a very loud way, is a security risk. Nagios warns you in the config file with a subtle:

*** ENABLING THIS OPTION IS A SECURITY RISK! ***

and then they ask you to change dont_blame_nrpe=0 to dont_blame_nrpe=1. Pretty subtle stuff. It suggests, rather mildly, that it is something you want to find every way to avoid.

Here’s what I did – on my main host, I defined a custom event handler for the service monitored by NRPE.

define service{
        use                             remote-service
        hostgroup_name                  my-prod-servers
        service_description             How hungry is the server?
        check_command                   check_nrpe!check_hunger_level
        event_handler                   my_event_handler!my_custom_hunger_command
        }

Then I defined the event handler command, my_event_handler, in commands.cfg

define command{
        command_name    my_event_handler
        command_line    $USER1$/eventhandlers/my_event_handler  -s $SERVICESTATE$ -t $SERVICESTATETYPE$ -a $SERVICEATTEMPT$ -H $HOSTADDRESS$ -c $ARG1$
        }

Notice that I pass the service state(-s), state type,(-t) service attempts(-a), hostname(-H) and command ($ARG1$, -c) as named arguments. Most importantly, I pass the hostname and the command. These, you’ll see, are used by the event handler to decide what to do. The script below, very largely the same as the one in Nagios’s documentation, I switched to bash (out of preference). It shows why we need the other arguments.

So, the script:

#!/bin/bash

#Uncomment the next two lines for debugging. Check logs in /tmp to see how execution's being done
#exec 2> /tmp/nagioslog."$$"
#set -x

# Event handler script for My Apps
# To run, use my_event_handler -s $SERVICESTATE$ -t $STATETYPE$ -a $SERVICEATTEMPT$ -H $HOSTADDRESS$ -c command

#

# Note: This script will only kick in if the service is

#       retried 3 times (in a "soft" state) or if the service somehow

#       manages to fall into a "hard" error state.

#

#function to display correct usage help
function usage(){
cat <<EOF
This script kicks in to perform remedial action on various services monitored by Nagios when their
state changes to near critical. It attempts to correct the issue so we don't go into a CRITICAL state
usage: my_event_handler -s <ServiceState> -t <StateType> -a <ServiceAttempts> -H <HostAddress> -c <Command>

OPTIONS:
  -s The service state (WARNING,UNKNOWN,CRITICAL)
  -t The state type (SOFT,HARD)
  -a The service attempts ( 1,2,3,4)
  -H The host address
  -c The command

EOF
}
serviceState=
stateType=
serviceAttempts=
hostname=
runCommand=

# What state is the service in?
while getopts "s:t:a:H:c:" OPTION
do
  case $OPTION in
        s)
          serviceState=$OPTARG
          ;;
        t)
          stateType=$OPTARG
          ;;
        a)
          serviceAttempts=$OPTARG
          ;;
        H)
          hostname=$OPTARG
          ;;
        c)
          runCommand=$OPTARG
          ;;
        ?)
          usage
          exit
          ;;
  esac
done

#Check that all the arguments have been provided
if [[ -z $serviceState ]] || [[ -z $stateType ]] || [[ -z $serviceAttempts ]] || [[ -z $hostname ]] || [[ -z $runCommand ]]
then
    usage
    exit 1
fi

case "$serviceState" in

        OK)

        # The service just came back up, so don't do anything...

        ;;

        WARNING)

        # Usually, we don't really care about warning states, since the service is probably still running...
        ##IF you have services for which you act even on warnings...
        if [[ $runCommand == "my_custom_hunger_command" ]] && [[ $serviceAttempts -gt 2 ]]
        then
                /usr/local/nagios/libexec/check_nrpe -H $hostname -c $runCommand
        fi
        ;;

        UNKNOWN)

        # We don't know what might be causing an unknown error, so don't do anything...

        ;;

        CRITICAL)

        # Aha!  The service appears to have a problem - perhaps we should restart the service...

        # Is this a "soft" or a "hard" state?

        case "$stateType" in

        # We're in a "soft" state, meaning that Nagios is in the middle of retrying the

        # check before it turns into a "hard" state and contacts get notified...

        SOFT)

                # What check attempt are we on?  We don't want to restart the service on the first

                # check, because it may just be a fluke!

                case "$serviceAttempts" in

                # Wait until the check has been tried 3 times before restarting the service.

                # If the check fails on the 4th time (after we restart the service), the state

                # type will turn to "hard" and contacts will be notified of the problem.

                # Hopefully this will restart the service successfully, so the 4th check will

                # result in a "soft" recovery.  If that happens no one gets notified because we

                # fixed the problem!

                3)

                        echo -n "Performing remedial action..."

                        # Call NRPE on the remote host

                        /usr/local/nagios/libexec/check_nrpe -H $hostname -c $runCommand

                        ;;

                        esac

                ;;

        # The  service somehow managed to turn into a hard error without getting fixed.

        # It should have been restarted by the code above, but for some reason it didn't.

        # Let's give it one last try, shall we?

        # Note: Contacts have already been notified of a problem with the service at this

        # point (unless you disabled notifications for this service)

        HARD)

                echo -n "Performing remedial action..."

                /usr/local/nagios/libexec/check_nrpe -H $hostname -c $runCommand

                ;;

        esac

        ;;

esac
exit 0

Then, in /usr/local/nagios/etc/nrpe.cfg on your remote host, add:

command[my_custom_hunger_command]=/usr/local/nagios/libexec/eventhandlers/my_custom_hunger_command

Lastly, restart NRPE on your remote host & Nagios on your main host.

Using this approach, you won’t have to send arguments to your remote hosts. The remedial command is called only when it needs to be since pre-checks are done.
Also, notice that you can use this event handler to monitor multiple services

define service{
        use                             remote-service
        hostgroup_name                  my-prod-servers
        service_description             Are the servers exercising?
        check_command                   check_nrpe!check_exercise_levels
        event_handler                   my_event_handler!my_custom_exercise_command
        }

In the event handler, $runCommand will be my_custom_exercise_command (and subsequently, the remote command called in NRPE is my_custom_hunger_command)

Note:
To troubleshoot, uncomment the lines shown in the comments in the script
If the remedial script you run on the remote host uses sudo, you’ll need to comment out #Defaults requiretty in /etc/sudoers on that host

Christmas inventions we need

If there’s ever a time in the year when we badly need inventions, it’s this one. All the merry making, travelling and binge-eating presents several opportunities to better mankind. It is documented that Einstein’s theory of relativity came to him after a hearty Christmas lunch. There he lay, battling sleep after the meal, unable to move a limb without using up whatever bit of energy he had left, when the idea appeared in his head. Unable to move towards the idea and grab it, he had to coax it into coming to him. On to the Christmas inventions:

Christmas Time Machine (CTM) – This fancy gadget, at your voice command, would take you to a Christmas back in the day. After a pilot study, the voice command would be removed because our accents would confuse it; CTM would be re-engineered to receive typed instructions. It would still take you to a Christmas of choice and pick you up right after lunch.

A fork that turns into a bed at the press of a button – This device, produced by Nokia of back in the day (to make it sturdy and long-lasting), would be used to eat your Christmas meal. After you are done, you’d press a button and it would transform into a bed that you’d collapse onto.

An application that locates for you a ‘real’ Christmas tree – This mobile application would give you specific directions to where to get a ‘real’ Christmas tree from. The ‘real’ trees are those that are cut from someone’s fence. At the tap of a button, you’d do negotiations on price (complete with pretend walking away in protest for the price to drop) and when you conclude, the tree would be delivered to you. You’d burn it amidst wild chants on New Year’s Eve.

Send food – This would work like mobile money except that you’d be sending and receiving food. There’d be an agent, obese from stealing from the business, to whom you’d give the food. He’d taste it when no one’s looking, pack it and then send it to the recipient using advanced technology.

Inventions aside, I hope you get a chance to wind down and enjoy this part of the year. The cheer’s great, the meaning of the season’s even greater. Joy to the world indeed. Happy Christmas, to you and yours.

When the people won’t let you go

Our First Deputy Prime Minister and Minister of Public Service, Henry Kajura last week let us in on how he’s under pressure to serve again as MP for Hoima Municipality. Fun fact, he’s been serving as a public servant, in one position or another, since 1966. That’s almost as long as the country has been independent. Just for perspective, the average life expectancy of a chicken is about 8 years. So he’s diligently served through about 7 chicken generations.

To be fair though, we probably didn’t get the entire story. One thing we weren’t told was, “What kind of pressure would someone who’s led for 50 years have to be under to be forced to stand again?” Let’s explore this to balance the reporting. This column is all about objectivity after all.

It is possible that everyone in Hoima has camped at his home and vowed not to leave till he promises to stand again. Put yourself in his shoes – having breakfast with all those people around you. Trying to watch your guilty pleasure, a telenovela, with them peering over your back doing running commentary. Biting into your mandazi as you sip tea in the evening as they collectively stare at you. I imagine he has to ask his kitchen staff to replace cereal with porridge, chicken with mukene and beans with katunkuma – you don’t want your constituents thinking you live too large. The Honorable under pressure probably can’t even use his flat screen TV or his iPad. There’s a movie script lurking in there – an honorable under siege. We could get someone like Morgan Freeman to play Kajura. Imagine the part where he falls asleep in parliament and Rebecca Kadaga (played by Kerry Washington) has to hurl a threat his way to wake him up.  He first hears the threat in his dream then it gets louder till he realizes it’s a real thing. Then a stand-off ensues, him vowing to sleep more, her baying for his neck. Pure box office gold.

Maybe the pressure is because one night, he came home to find a guy in a suit and shades, holding a bow and arrow, who promised to be back should he refuse to stand.

All pressure aside, maybe, like our beloved leader, he’s the only one with a vision. Hoima cannot live without you. After 50 years in leadership, surely, you have a lot to give. Sleeping in parliament is only because you have big dreams for the nation, and those dreams come to you easier well, when you are asleep.

Why Spain is not Uganda

Last week, the Guardian, a newspaper that’s a big deal in the UK, run a poll on which holiday destination was more attractive to their readers – Uganda or Spain. Being the buxom beauty that the pearl is, we beat Spain black and white in the poll. Some analysts put it down to Spaniards not knowing that the poll was going on since it was conducted in English. If the poll question had been “Uganda o España : donde preferirías ir de vacaciones ?”, Spain would have had a running chance at winning it. Since the analysts who said this had a foreign accent, we couldn’t quite argue with them. For good measure though, I asked a few others to hear their opinion – why did Spain get beaten so badly? Should they pack up and move to Uganda? Is there a wail loud enough to capture the anguish they are currently going through? Was it Uganda’s stunning beauty? Was the fight unfair to start with? Are we really that awesome? Over to the analysts.

Some said that the internet in Spain was down during the poll. They, after one phone call and a Google search, stated that the guy with the key to the store that has the internet in Spain had gone off to play soccer with his friends. He’s trying out for Atletico Astorgaand and if he makes it, he’ll have to choose whether to hand the key to the internet to his son (who’s still crawling) or let his own soccer dream die.

One excited analyst, who kept dozing off during the interview, said that the Spaniards were out watching a hugely popular Telenovela. The show was at that critical part where Alejandro and Maria, who are madly in love, were just about to discover that they are actually brother and sister.

 Some voices from Spain were heard asking for a re-count. They said that being a former British protectorate, Uganda had manipulated the UK paper’s vote in its favour. “They stuffed the ballot boxes those people,” said one angry Spaniard. He said it in Spanish though and with a lot of gestures. He didn’t respond when it was pointed out that the poll was conducted online.

One analyst, wearing a yellow T-shirt, said that our beloved leader was behind the win. He didn’t clarify how this was even possible.

Most of the analysts put it down to Uganda actually being awesome; very warm people, lovely weather, stunning natural beauty and a huge variety of wild and bird life. I agree with this lot.

Hit song to our beloved leader

I particularly love this article because on a dreary evening working late, my parents called me, laughing heartily and quoting lines from it, saying it had totally, totally cracked them up.  I hope you enjoy it too

Kikyusa Sub-county chairperson, Abubaker Mubiru Ssematimba, last week asked schools to compose Museveni-praise songs. I feel for the young mind hunched over his Picfare book, writing and angrily throwing away draft after draft of lyrics. Coining the perfect lyrics to show gratitude to anyone is no mean-feat, especially someone as larger-than-life as our leader.

Here, here young one, let’s do this together. Let’s work on this so you can go back to focusing on passing your examinations, ok? Before you get too excited though, I have no experience writing songs. True story. To be fair though, I share a surname with the late composer of our national anthem-surely, that counts for something. Now that your faith is restored, let’s proceed.

I propose that you perform this song when our beloved leader visits your school (or when your school gets a chance to sing at Kololo at the Independence/Heroes/Liberation/other day celebrations)

 (Beloved leader walks, while waving to the gathered crowd, towards the assembled children)

(Single child steps to the front, hands him a huge bouquet of flowers – no roses because they have thorns and we could charge you with treason, young one)

(Single child walks backwards towards the rest of the choir, not turning his back to the leader)

(Single child breaks into song, two steps away from the rest)

You are welcome dear leader, you are welcome
We are happy to receive you
You are welcome
You are….

(The rest jump in, chirpy, smiley, excited)

You are welcome dear leader
We thank you today
(They start to do a hand and foot movements in sync with the lyrics, feet swinging forward while the hands go back, alternate, repeat)

Where would we be without you
We would not exist
Because our folks would have gone with heart disease
What with all the sausages they were swallowing

(Energetic young girls break out into Bakisimba, move in single file from the back of the choir to the front then split into four sets and break into a heavy routine)

With you here, we swell with excitement
Joy is our indictment
Mobile phones, civilization, soda, TV, clothes you brought us
Those naysayers probably can’t even see KFC
Our folks now even sleep at night,
Actually they don’t since they now have discos
Long life, prosperity, to you and yours is our wish
You are welcome dear leader (repeat till fade)

(Acrobatic young boys somersault to where the four sets of dancing girls are and together, dance off stage, smiley, chirpy, happy)