check_job_overtime

Look for jobs that are running past their available walltime

Note: it is assumed that TORQUE is used as a resource manager and Maui as a scheduler. Tested with TORQUE 2.3.5 and Maui 3.2.6p21.

Requires a valid showq Maui client.

Usage

check_job_overtime

This plugin has no command line parameters; showq is executed locally.

Note that it can be quite normal for a job to run a few minuted beyond its scheduled walltime, so if you plan to use an event handler to kill runaway jobs, be sure to check multiple times at suitable intervals this plugin!

#!/usr/bin/perl -w use strict ; use lib "/opt/nagios/libexec" ; use utils qw($TIMEOUT %ERRORS &print_revision &support) ; use vars qw($PROGNAME) ; use Getopt::Long ; $PROGNAME = "check_job_overtime" ; my $showq = "/opt/nagios/libexec/external/showq" ; unless ( -x $showq ) { print "UNKNOWN - $showq not found\n" ; exit $ERRORS{'UNKNOWN'} ; } my $badjobs ; my $badjobs_count = 0 ; open RUNNING, "$showq -r |" or exit $ERRORS{'UNKNOWN'} ; while ( <RUNNING> ) { chomp ; next unless m/^\s+(\d+).\sR\s+[a-zA-Z]+\s+[\d.]+\s+[\d.]+\s+[a-zA-Z]+\s+([a-z]+)\s+[a-z]+\s+[a-z\d]+\s+[\d]+\s+-([\d:]+)/ ; $badjobs_count ++ ; $badjobs .= " $1/$2/$3" ; } if ( $badjobs_count > 0 ) { my $job_string = ( $badjobs_count > 1 ? 'jobs are' : 'job is' ) ; print "WARNING - $badjobs_count $job_string in overtime|$badjobs\n" ; exit $ERRORS{'WARNING'} ; } print "OK\n" ; exit $ERRORS{'OK'} ;
download code