NAME
check_jmx4perl - Nagios plugin using jmx4perl for accessing JMX data remotely
SYNOPSIS
# Check for used heap memory (absolute values)
check_jmx4perl --url http://localhost:8888/jolokia \
--name memory_used \
--mbean java.lang:type=Memory \
--attribute HeapMemoryUsage \
--path used \
--critical 10000000 \
--warning 5000000
# Check that used heap memory is less than 80% of the available memory
check_jmx4perl --url http://localhost:8888/jolokia \
--alias MEMORY_HEAP_USED \
--base MEMORY_HEAP_MAX \
--critical :80
# Use predefined checks in a configuration file with a server alias
# Server alias is 'webshop', check is about requests per minute for the
# servlet 'socks_shop'
check_jmx4perl --config /etc/nagios/check_jmx4perl/tomcat.cfg
--server webshop \
--check tc_servlet_requests \
--critical 1000 \
socks_shop
# Check for string values by comparing them literally
check_jmx4perl --url http://localhost::8888/jolokia \
--mbean myDomain:name=myMBean \
--attribute stringAttribute \
--string \
--critical 'Stopped' \
--warning '!Started'
# Check that no more than 5 threads are started in a minute
check_jmx4perl --url http://localhost:8888/jolokia \
--alias THREAD_COUNT_STARTED \
--delta 60 \
--critical 5
# Execute a JMX operation on an MBean and use the return value for threshold
# Here a thread-deadlock is detected.
check_jmx4perl --url http://localhost:8888/jolokia \
--mbean java.lang:type=Threading \
--operation findDeadlockedThreads \
--null no-deadlock \
--string \
--critical '!no-deadlock' \
--critical 10
# Use check_jmx4perl in proxy mode
check_jmx4perl --url http://localhost:8888/jolokia \
--alias MEMORY_HEAP_USED \
--critical 10000000 \
--target service:jmx:rmi:///jndi/rmi://bhut:9999/jmxrmi
DESCRIPTION
check_jmx4perl
is a Nagios plugin for monitoring Java applications. It uses an agent based approach for accessing JMX exposed information remotely.
Before start using check_jmx4perl
an agent must be installed on the target platform. For JEE application server this is a simple webapplication packaged as a war
archive. For other platforms, other agents are available, too. Please refer to the README
for installation instructions and the supported platforms.
check_jmx4perl
s can also be used in an agentless mode (i.e. no agent needs to be installed on the target platform). See "Proxy mode" for details.
This plugin can be configured in two ways: Either, all required parameters for identifying the JMX information can be given via the command line. Or, a configuration file can be used to define one or more Nagios checks. This is the recommended way, since it allows for more advanced features not available when using the command line alone. Each command line argument has an equivalent option in the configuration files, though.
This documentation contains four parts. First, a tutorial gives a 5 minute quickstart for installing and using check_jmx4perl
. The middle part offers some technical background information on JMX itself, the features provided by this plugin and finally the command line arguments and the configuration file directives are described.
TUTORIAL
Before we dive into the more nifty details, this 5 minutes quickstart gives a simple cooking recipe for configuration and setup of check_jmx4perl
.
This tutorial uses tomcat as an application server. Download it from http://tomcat.apache.org (either version 5 or 6) and extract it:
$ tar zxvf apache-tomcat-*.tar.gz $ # We need this variable later on: $ TC=`pwd`/apache-tomcat*
Download jmx4perl from http://search.cpan.org/~roland/jmx4perl and install it:
$ tar zxvf jmx4perl-*.tar.gz $ cd jmx4perl* $ # Store current directory for later reference: $ J4P=`pwd` $ perl Build.PL $ sudo ./Build install
This is installs the Perl modules around
JMX::Jmx4Perl
which can be used for programmatic JMX access. There are some CPAN dependencies for jmx4perl, the build will fail if there are missing modules. Please install the missing modules via cpan (cpan module
). The Nagios plugincheck_jmx4perl
is installed in a standard location (/usr/bin, /usr/local/bin or whatever your Perl installation thinks is appropriate) as well as the other scriptsjmx4perl
(a generic tool for accessing JMX) andj4psh
(an interactive JMX shell).Deploy the Jolokia agent in Tomcat:
$ cd $TC/webapps $ jolokia
Start Tomcat:
$ $TC/bin/startup.sh
Check your setup:
$ jmx4perl http://localhost:8080/jolokia
This prints out a summary about your application server. http://localhost:8080/jolokia is the URL under which the agent is reachable. Tomcat itself listens on port 8080 by default, and any autodeployed war archive can be reached under its filename without the .war suffix (jolokia in this case).
Try a first Nagios check for checking the amount of available heap memory in relation to the maximal available heap:
$ check_jmx4perl --url http://localhost:8080/jolokia \ --mbean java.lang:type=Memory \ --attribute HeapMemoryUsage \ --path used \ --base java.lang:type=Memory/HeapMemoryUsage/max \ --warning 80 \ --critical 90 OK - [java.lang:type=Memory,HeapMemoryUsage,used] : In range 9.83% (12778136 / 129957888) | '[java.lang:type#Memory,HeapMemoryUsage,used]'=12778136;103966310.4;116962099.2;0;129957888
where
- --url http://localhost:8080/jolokia
-
is the agent URL
- --mbean java.lang:type=Memory
-
is the MBean name
- --attribute HeapMemoryUsage
-
is the attribute to monitor
- --path used
-
is an inner path (see "Paths"), which specifies an inner value within a more complex structure. The value
HeapMemoryUsage
is a composed value (Jav type: CompositeData) which combines multiple memory related data. The complete value can be viewed with jmx4perl:$ jmx4perl http://localhost:8080/jolokia read java.lang:type=Memory HeapMemoryUsage { committed => 85000192, init => 0 max => 129957888, used => 15106608, }
- --base java.lang:type=Memory/HeapMemoryUsage/max
-
is the base value for which a relative threshold should be applied. This is a shortcut notation in the format mbean
/
attribute/
path. - --warning 80
-
is the warning threshold in percent. I.e. a
WARNING
will be raised by this plugin when the heap memory usage is larger than 80% of the maximal available heap memory for the application server (which is smaller than the available memory of the operating system) - --critical 90
-
is the critical threshold in percent. If the available heap memory reaches 90% of the available heap, a
CRITICAL
alert will be returned.
All available command line options are described in "COMMAND LINE".
For more complex checks the usage of a configuration file is recommended. This also allows you to keep your Nagios service definitions small and tidy. E.g. for monitoring the number of request per minute for a certain web application, a predefined check is available:
$ check_jmx4perl --url http://localhost:8080/jolokia \ --config $J4P/config/tomcat.cfg \ --critical 100 \ --check tc_servlet_requests \ jolokia-agent OK - 15.00 requests/minute | 'Requests jolokia-agent'=15;5000;100
where
- --config $J4P/config/tomcat.cfg
-
is the path to configuration file. There a several predefined checks coming with this distribution, which are documented inline. Look there for some inspiration for what to check.
- --critical 100
-
A threshold von 100, i.e. the checked value must be 100 or less, otherwise a critical alert is raised.
- --check tc_servlet_requests
-
is the name of the check to perform which must be defined in the configuration file
- jolokia-agent
-
is an extra argument used by the predefined check. It is the name of the servlet for which the number of requests should be monitored. To get the name of all registered servlets use
jmx4perl list
:$ jmx4perl http://localhost:8080/jolokia list | grep j2eeType=Servlet
The servlet name is the value of the
name
property of the listed MBeans.
Configuration files are very powerful and are the recommended way for configuring
check_jmx4perl
for any larger installation. Features like multi checks are even only available when using a configuration file. The syntax for configuration files are explained in depth in "CONFIGURATION".Finally, a Nagios service definition needs to be added. For the memory example above, a command for relative checks can be defined:
define command { command_name check_jmx4perl_relative command_line $USER3$/check_jmx4perl \ --url $ARG1$ \ --mbean $ARG2$ \ --attribute $ARG3$ \ --path $ARG4$ \ --base $ARG5$ \ $ARG6$ }
Put this into place where you normally define commands (either in the global Nagios commands.cfg or in a specific commands configuration file in the commands directory).
$USER3
is a custom variable and should point to the directory wherecheck_jmx4perl
is installed (e.g. /usr/local/bin).The service definition itself then looks like:
define service { service_description j4p_localhost_memory host_name localhost check_command check_jmx4perl_relative \ !http://localhost:8080/jolokia \ !java.lang:type=Memory \ !HeapMemoryUsage \ !used \ !java.lang:type=Memory/HeapMemoryUsage/max \ !--warning 80 --critical 90 }
Add this section to your service definitions (depending on your Nagios installation). This example adds a service to host
localhost
for checking the heap memory, raising aWARNING
if 80% of the available heap is used and aCRITICAL
if more than 90% of the heap memory is occupied.
Installing and using jmx4perl is really that easy. The Nagios configuration in this example is rather simplistic, of course a more flexible Nagios setup is possible. The blog post http://labs.consol.de//blog/jmx4perl/check_jmx4perl-einfache-servicedefinitionen/ (written by Gerhard Lausser) shows some advanced configuration setup. (It is in german, but the automatic translation from http://bit.ly/bgReAs seems to be quite usable).
REFERENCE
This section explains the JMX basics necessary to better understand the usage of check_jmx4perl
. It tries to be as brief as possible, but some theory is required to get the link to the Java world.
MBeans
JMX's central entity is an MBean
. An MBean exposes management information in a well defined way. Each MBean has a unique name called Object Name with the following structure:
domain:attribute1=value1,attribute2=value2, .....
E.g.
java.lang:type=Memory
points to the MBean which lets you access the memory information of the target server.
Unfortunately, except for so called MXBeans (http://java.sun.com/j2se/1.5.0/docs/api/java/lang/management/package-summary.html) there is no standard naming for MBeans. Each platform uses its own. There used to be a naming standard defined in JSR77 (http://jcp.org/en/jsr/detail?id=77), unfortunately it was never widely adopted.
There are various ways for identifying MBeans on a server:
Use
jmx4perl --list
to list all registered MBeans. In additionjmx4perl --attributes
dumps out all known MBean attributes along with their values. (Be careful, the output can be quite large)Use
j4psh
for interactively exploring the JMX namespace.Use an alias. An alias is a shortcut for an MBean name, predefined by JMX::Jmx4Perl. All known aliases can be shown with
jmx4perl aliases
. Since each platform can have slightly different MBean names for the same information, this extra level of indirection might help in identifying MBeans. See "Aliases" for more about aliases.Use a predefined check.
check_jmx4perl
comes with quite some checks predefined in various configuration files. These are ready for use out of the box. "Predefined checks" are described in an extra section.Ask your Java application development team for application specific MBean names.
Attributes and Operations
check_jmx4perl
can obtain the information to monitor from two sources: Either as MBean attributes or as a return value from JMX operations. Since JMX values can be any Java object, it is important to understand, how check_jmx4perl
(or jmx4perl in general) handles this situation.
Simple data types can be used directly in threshold checking. I.e. the following data types can be used directly
Integer
Long
Float
Double
Boolean
String
String
and Boolean
can be used in string checks only, whereas the others can be used in both, numeric and string checks (see "String checks").
For numeric checks, the threhsholds has to be specified according to the format defined in http://nagiosplug.sourceforge.net/developer-guidelines.html#THRESHOLDFORMAT
Paths
For more complex types, check_jmx4perl
provides the concept of so called paths for specifying an inner attribute of a more complex value. A path contains parts separated by slashes (/). It is similar to an XPath expression for accessing parts of an XML document. Each part points to an inner level of a complex object.
For example, the MBean java.lang:type=Memory
exposes an attribute called HeapMemoryUsage
. This attribute is a compound data type which contains multiple entries. Looking with jmx4perl
at this attribute
$ jmx4perl http://localhost:8080/jolokia read java.lang:type=Memory HeapMemoryUsage
{
committed => 85000192,
init => 0
max => 129957888,
used => 15106608,
}
it can be seen, that there are 4 values coming with the reponse. With a path used
one can directly pick the used heap memory usage (8135440 bytes in this case) which then can be used for a threshold check.
$ check_jmx4perl --url http://localhost:8080/jolokia \
--mbean java.lang:type=Memory \
--attribute HeapMemoryUsage \
--path used \
--critical 100000000
OK - [java.lang:type=Memory,HeapMemoryUsage,used] : Value 10136056 in range | ...
Attributes
Attributes are values obtained from MBean properties. Complex values are translated into a JSON structure on the agent side, which works for most types. To access a single value from a complex value, the path mechanism described above can be used. Thresholds can be applied to simple data types only, so for complex attributes a path is required.
Operations
The return values of operations can be used for threshold checking, too. Since a JMX exposed operation can take arguments, these has to be provided as extra arguments on the command line or in the configuration via the Args
configuration directive. Due to the agent's nature and the protocol used (JSON), only simple typed arguments like strings, numbers or booleans ("true"/"false") can be used.
Example:
$ check_jmx4perl --url http://localhost:8888/jolokia \
--mbean jolokia:type=Runtime \
--operation getNrQueriesFor \
--critical 10 \
"operation" \
"java.lang:type=Memory" \
"gc"
This example contacts a MBean jolokia:type=Runtime
registered by the jolokia agent in order to check for the number of queries for a certain MBean via this agent. For this purpose an JMX operation getNrQueriesFor
is exposed which takes three arguments: The type ("operation"/"attribute"), the MBean's ObjectName and the operation/attribute name which was called.
If the operation to be called is an overloaded operation (i.e. an operation whose name exists multiple times on the same MBean but with different parameter types), the argument types must be given within parentheses:
--operation checkUserCount(java.lang.String,java.lang.String)
Aliases
Aliases are shortcut for common MBean names and attributes. E.g. the alias MEMORY_HEAP_MAX
specifies the MBean java.lang:type=Memory
, the attribute HeapMemoryUsage
and the path max
. Aliases can be specified with the --alias
option or with the configuration directive Alias
. Aliases can be translated to different MBean names on different application server. For this check_jmx4perl
uses an autodetection mechanism to determine the target platform. Currently this mechanism uses one or more extra server round-trips. To avoid this overhead, the --product
option (configuration: Product
) can be used to specify the target platform explicitely. This is highly recommended in case you are using the aliasing feature.
Aliases are not extensible and can not take any parameters. All availables aliases can be viewed with
jmx4perl aliases
A much more flexible alternative to aliases are parameterized checks, which are defined in a configuration file. See "CONFIGURATION" for more details about parameterized checks.
Relative Checks
Relative values are often more interesting than absolute numbers. E.g. the knowledge that 140 MBytes heap memory is used is not as important as the knowledge, that 56% of the available memory is used. Relative checks calculate the ratio of a value to a base value. (Another advantage is that Nagios service definitions for relative checks are generic as they can be applied for target servers with different memory footprints).
The base value has to be given with --base
(configuration: Base
). The argument provided here is first tried as an alias name or checked as an absolute, numeric value. Alternatively, you can use a full MBean/attribute/path specification by using a /
as separator, e.g.
... --base java.lang:type=Memory/HeapMemoryUsage/max ...
If one of these parts (the path is optional) contains a slash within its name, the slash must be escaped with a backslash (\/). Backslashes in MBean names are escaped with a double backslash (\\).
Alternatively --base-mbean
, --base-attribute
and --base-path
can be used to specify the parts of the base value separately.
Example:
check_jmx4perl --url http://localhost:8080/jolokia \
--value java.lang:type=Memory/HeapMemoryUsage/used \
--base java.lang:type=Memory/HeapMemoryUsage/max \
--critical 90
check_jmx4perl --url http://localhost:8080/jolokia \
--value java.lang:type=Memory/HeapMemoryUsage/used \
--base-mbean java.lang:type=Memory \
--base-attribute HeapMemoryUsage \
--base-path max \
--critical 90
This check will trigger a state change to CRITICAL if the used heap memory will exceed 90% of the available heap memory.
Incremental Checks
For some values it is worth monitoring the increase rate (velocity). E.g. for threads it can be important to know how fast threads are created.
Incremental checks are switched on with the --delta
option (configuration: Delta
). This option takes an optional argument which is interpreted as seconds for normalization.
Example:
check_jmx4perl --url http://localhost:8080/jolokia \
--mbean java.lang:type=Threading \
--attribute TotalStartedThreadCount \
--delta 60 \
--critical 5
This will fail as CRITICAL if more than 5 threads are created per minute (60 seconds). Technically check_jmx4perl
uses the history feature of the jolokia agent deployed on the target server. This will always store the result and the timestamp of the last check on the server side and returns these historical values on the next check so that the velocity can be calculated. If no value is given for --delta
, no normalization is used. In the example above, without a normalization value of 60, a CRITICAL is returned if the number of threads created increased more than 5 between two checks.
--delta
doesn't work yet with --base
(e.g. incremental mode for relative checks is not available).
String checks
In addition to standard numerical checks, direct string comparison can be used. This mode is switched on either explicitely via --string
(configuration: String
) or by default implicitely if a heuristics determines that a value is non-numeric. Numeric checking can be enforced with the option --numeric
(configuration: Numeric).
For string checks, --critical
and --warning
are not treated as numerical values but as string types. They are compared literally against the value retrieved and yield the corresponding Nagios status if matched. If the threshold is given with a leading !
, the condition is negated. E.g. a --critical '!Running'
returns CRITICAL
if the value not equals to Running
. Alternatively you can also use a regular expression by using qr/.../
as threshold value (substitute ...
with the pattern to used for comparison). Boolean values are returned as true
or false
strings from the agent, so you can check for them as well with this kind of string comparison.
No performance data will be generated for string checks by default. This can be switched on by providing --perfdata on
(or "PerfData on
" in the configuration). However, this probably doesn't make much sense, though.
Output Tuning
The output of check_jmx4perl
can be highly customized. A unit-of-measurement can be provided with the option --unit
(configuration: Unit
) which specifies how the the attribute or an operation's return value should be interpreted. The units available are
B - Byte
KB - Kilo Byte
MB - Mega Byte
GB - Giga Byte
TB - Terra Byte
us - Microseconds
ms - Milliseconds
s - Seconds
m - Minutes
h - Hours
d - Days
The unit will be used for performance data as well as for the plugin's output. Large numbers are converted to larger units automatically (and reverse for small number that are smaller than 1). E.g. 2048 KB
is converted to 2 MB
. Beautifying by conversion is only performed for the plugin output, not for the performance data for which no conversions happens at all.
Beside unit handling, you can provide your own label for the Nagios output via --label
. The provided option is interpreted as a pattern with the following placeholders:
%v the absolute value
%f the absolute value as floating point number
%r the relative value as percentage (--base)
%q the relative value as ratio of value to base (--base)
%u the value's unit for the output when --unit is used (after shortening)
%w the base value's unit for the output when --unit is used (after shortening)
%b the absolut base value as it is used with --base
%c the Nagios exit code in the Form "OK", "WARNING", "CRITICAL"
or "UNKNOWN"
%t Threshold value which failed ("" when the check doesn't fail)
%n name, either calulated automatically or given with --name
%d the delta value used for normalization when using incremental mode
%y WARNING threshold as configured
%z CRITICAL threshold as configured
Note that %u
and %w
are typically not the same as the --unit
option. They specify the unit after the conversion for the plugin output as described above. You can use the same length modifiers as for sprintf
to fine tune the output.
Example:
check_jmx4perl --url http://localhost:8888/jolokia \
--alias MEMORY_HEAP_USED \
--base MEMORY_HEAP_MAX \
--critical :80 \
--label "Heap-Memory: %.2r% used (%.2v %u / %.2b %w)" \
--unit B
will result in an output like
OK - Heap-Memory: 3.48% used (17.68 MB / 508.06 MB) | '[MEMORY_HEAP_USED]'=3.48%;;:80
Security
Since the jolokia-agent is usually a simple war-file, it can be secured as any other Java Webapplication. Since setting up authentication is JEE Server specific, a detailed instruction is beyond the scope of this document. Please refer to your appserver documentation, how to do this. At the moment, check_jmx4perl
can use Basic-Authentication for authentication purposes only.
In addition to this user/password authentication, the jolokia-agent uses a policy file for fine granular access control. The policy is defined with an XML file packaged within the agent. In order to adapt this to your needs, you need to extract the war file, edit it, and repackage the agent with a policy file. A future version of jmx4perl might provide a more flexible way for changing the policy.
In detail, the following steps are required:
Download jolokia.war and a sample policy file jolokia-access.xml into a temporary directory:
$ jolokia $ jolokia --policy
Edit the policy according to your needs.
$ vi jolokia-access.xml
Repackage the war file
$ jolokia repack --policy jolokia.war
Deploy the agent jolokia.war as usual
The downloaded sample policy file jolokia-access.xml contains inline documentation and examples, so you can easily adapt it to your environment.
Restrictions can be set to on various parameters :
Client IP address
Access to the jolokia-agent can be restricted based on the client IP accessing the agent. A single host, either with hostname or IP address can be set or even a complete subnet.
Example:
<remote>
<host>127.0.0.1</host>
<host>10.0.0.0/16</host>
</remote>
Only the localhost or any host in the subnet 10.0 is allowed to access the agent. If the <remote>
section is missing, access from all hosts is allowed.
Commands
The access can be restricted to certain commands.
Example:
<commands>
<command>read</command>
</commands>
This will only allow reading of attributes, but no other operation like execution of operations. If the <commands>
section is missing, any command is allowed. The commands known are
- read
-
Read an attribute
- write
-
Write an attribute (used by
check_jmx4perl
only when using incremental checks) - exec
-
Execution of an operation
- list
-
List all MBeans (not used by
check_jmx4perl
) - version
-
Version command (not used by
check_jmx4perl
) - search
-
Search for MBean (not used by
check_jmx4perl
)
Specific MBeans
The most specific policy can be put on the MBeans themselves. For this, two sections can be defined, depending on whether a command is globaly enabled or denied.
- <allow>
-
The
<allow>
section is used to switch on access for operations and attributes in caseread
,write
orexec
are globally disabled (see above). Wildcards can be used for MBean names and attributes/and operations.Example:
<allow> <mbean> <name>jolokia:*</name> <operation>*</operation> <attribute>*</attribute> </mbean> <mbean> <name>java.lang:type=Threading</name> <operation>findDeadlockedThreads</operation> </mbean> <mbean> <name>java.lang:type=Memory</name> <attribute mode="read">Verbose</attribute> </mbean> </allow>
This will allow access to all operation and attributes of all MBeans in the
jolokia:
domain and to the operationfindDeadlockedThreads
on the MBeanjava.lang:type=Threading
regardless whether theread
orexec
command is enabled globally. The attributeVerbose
onjava.lang:type=Memory
is allowed to be read, but cannot be written (if themode
attribute is not given, both read and write is allowed by default). - <deny>
-
The
<deny>
section forbids access to certain MBean's operation and/or attributes, even when the command is allowed globally.Example:
<deny> <mbean> <!-- Exposes user/password of data source, so we forbid this one --> <name>com.mchange.v2.c3p0:type=PooledDataSource*</name> <attribute>properties</attribute> </mbean> </deny>
This will forbid the access to the specified attribute, even if
read
is allowed globally. If there is an overlap between <allow> and <deny>, <allow> takes precedence.
Proxy mode
check_jmx4perl
can be used in an agentless mode as well, i.e. no jolokia-agent needs to deployed on the target server. The setup for the agentless mode is a bit more complicated, though:
The target server needs to export its MBeans via JSR-160. The configuration for JMX export is different for different JEE Server. http://labs.consol.de has some cooking recipes for various servers (JBoss, Weblogic).
A dedicated proxy server needs to be setup on which the jolokia.war gets deployed. This can be a simple Tomcat or Jetty servlet container. Of course, an already existing JEE Server can be used as proxy server as well.
For using
check_jmx4perl
the target JMX URL for accessing the target server is required. This URL typically looks likeservice:jmx:rmi:///jndi/rmi://host:9999/jmxrmi
but this depends on the server to monitor. Please refer to your JEE server's documentation for how the export JMX URL looks like.
check_jmx4perl
uses the proxy mody if the option--target
(configuration: <Target>) is provided. In this case, this Nagios plugin contacts the proxy server specified as usual with--url
(config: Url in Server section) and put the URL specified with--target
in the request. The agent in the proxy then dispatches this request to the real target and uses the JMX procotol specified with in the target URL. The answer received is then translated into a JSON response which is returned tocheck_jmx4perl
.Example:
check_jmx4perl --url http://proxy:8080/jolokia \ --target service:jmx:rmi:///jndi/rmi://jeeserver:9999/jmxrmi --alias MEMORY_HEAP_USED --base MEMORY_HEAP_MAX --critical 90
Here the host proxy is listening on port 8080 for jolokia requests and host jeeserver exports its JMX data via JSR-160 over port 9999. (BTW, proxy can be monitored itself as usual).
So, what mode is more appropriate ? Both, the agent mode and the proxy mode have advantages and disadvantages.
Advantages
No agent needs to be installed on the target server. This might be useful for policy reasons.
Compared to other Nagios JMX plugin's no JVM startup is required since the proxy server is already running.
Disadvantages
It takes two hops to get to the target server
Exporting JMX via JSR-160 is often not that easy as it may seem. (See post series on remote JMX on labs.consol.de)
Some features like merging of MBean Servers are not available in proxy mode. (i.e you need to know in advance which MBean-Server on the target you want to contact for a certain MBean, since this information is part of the JMX URL)
Bulk request needs to be detangled into multiple JMX request since JSR-160 doesn't know anything about bulk requests.
jmx4perl's fine granular security policy is not available, since JSR-160 JMX is an all-or-nothing thing. (except you are willing to dive deep into Java Security stuff)
For JSR-160 objects to be transferable to the proxy, the proxy needs to know about the Java types and those types must be serializable. If this is not the case, the proxy isn't able to collect the information from the target. So only a subset of MBeans can be monitored this way.
The agent protocol is more flexible since it translates the data into a JSON structure before putting it on the wire.
To summarize, I would always recommend the agent mode over the proxy mode except when an agentless operation is required (e.g. for policy reasons).
COMMAND LINE
The pure command line interface (without a configuration file) is mostly suited for simple checks where the predefined defaults are suitable. For all other use cases, a configuration file fits better.
check_jmx4perl
knows about the following command line options:
- --url (-u)
-
The URL for accessing the target server (or the jolokia-proxy server, see "Proxy Mode" for details about the JMX proxy mode)
Example:
--url http://localhost:8080/jolokia
- --mbean (-m)
-
Object name of MBean to access
Example:
--mbean java.lang:type=Runtime
- --attribute (-a)
-
A MBean's attribute name. The value of this attribute is used for threshold checking.
Example:
--attribute Uptime
- --operation (-o)
-
A MBean's operation name. The operation gets executed on the server side and the return value is used for threshold checking. Any arguments required for this operation has to be given as additional arguments to
check_jmx4perl
. See "Attributes and Operations" for details.Example:
check_jmx4perl ... --mbean java.lang:type=Threading \ --operation getThreadUserTime 1
Operation
getThreadUserTime
takes a single argument the thread id (a long) which is given as extra argument. - --path (-p)
-
Path for extracting an inner element from an attribute or operation return value. See "Paths" for details about paths.
Example:
--path used
- --value
-
Shortcut for giving
--mbean
,--attribute
and--path
at once.Example:
--value java.lang:type=Memory/HeapMemoryUsage/used
Any slash (/) in the MBean name must be escaped with a backslash (\/). Backslashes in names has to be escaped as \\.
- --base (-b)
-
Switches on relative checking. The value given points to an attribute which should be used as base value and has to be given in the shortcut notation described above. Alternatively, the value can be an absolute number or an alias name ("Aliases") The threshold are the interpreted as relative values in the range [0,100]. See "Relative Checks" for details.
Example:
--base 100000 --base java.lang:type=Memory/HeapMemoryUsage/max --base MEMORY_HEAP_MAX
- --delta (-d)
-
Switches on incremental checking, i.e. the increase rate (or velocity) of an attribute or operation return value is measured. The value given here is used for normalization (in seconds). E.g.
--delta 60
normalizes the velocity to 'growth per minute'. See "Incremental Checks" for details. - --string
-
Forces string checking, in which case the threshold values are compared as strings against the measured values. See "String checks" for more details. By default, a heuristic based on the measured value is applied to determine, whether numerical or string checking should be use
Example:
--string --critical '!Running'
- --numeric
-
Forces numeric checking, in which case the measured valued are compared against the given thresholds according to the Nagios developer guideline specification (http://nagiosplug.sourceforge.net/developer-guidelines.html#THRESHOLDFORMAT)
Example:
--numeric --critical ~:80
- --null
-
The value to be used in case the attribute or the operation's return value is
null
. This is useful when doing string checks. By default, this value is "null
".Example:
--null "no deadlock" --string --critical "!no deadlock"
- --name (-n)
-
Name to be used for the performance data. By default a name is calculated based on the MBean's name and the attribute/operation to monitor.
Example:
--name "HeapMemoryUsage"
- --label (-l)
-
Label for using in the plugin output which can be a format specifier as described in "Output Tuning".
Example:
--label "%.2r% used (%.2v %u / %.2b %w)"
- --perfdata
-
Switch off ("off") or on ("on") performance data generation. Performance data is generated by default for numerical checks and omitted for string based checks. For relative checks, if the value is '%' then performance data is appended as relative values instead of absolute values.
- --unit
-
Natural unit of the value measured. E.g. when measuring memory, then the memory MXBean exports this number as bytes. The value given here is used for shortening the value's output by converting to the largest possible unit. See "Output Tuning" for details.
Example:
--alias MEMORY_HEAP_USED --unit B
- --critical (-c)
-
Critical threshold. For string checks, see "String checks" for how the critical value is interpreted. For other checks, the value given here should conform to the specification defined in http://nagiosplug.sourceforge.net/developer-guidelines.html#THRESHOLDFORMAT.
Example:
--critical :90
- --warning (-w)
-
Warning threshold, which is interpreted the same way as the
--critical
threshold (see above). At least a warning or critical threshold must be given. - --alias
-
An alias is a shortcut for an MBean attribute or operation. See "Aliases" for details.
Example:
--alias RUNTIME_UPTIME
- --product
-
When aliasing is used,
check_jmx4perl
needs to known about the target server type for resolving the alias. By default it used an autodetection facility, which at least required an additional request. To avoid this, the product can be explicitely specified hereExample:
--product jboss
- --user, --password
-
User and password needed when the agent is secured with Basic Authentication. By default, no authentication is used.
- --timeout (-t)
-
How long to wait for an answer from the agent at most (in seconds). By default, the timeout is 180s.
- --method
-
The HTTP metod to use for sending the jmx4perl request. This can be either
get
orpost
. By default, an method is determined automatically.get
for simple, single requests,post
for bulk request or requests using a JMX proxy. - --proxy
-
A HTTP proxy server to use for accessing the jolokia-agent.
Example:
--proxy http://proxyhost:8001/
- --legacy-escape
-
When the deployed Jolokia agent's version is less than 1.0, then this option should be used since the escape scheme as changed since version 1.0. This option is only important for MBeans whose names contain slashes. It is recommended to upgrade the agent to a post 1.0 version, though.
- --target, --target-user, --target-password
-
Switches on jolokia-proxy mode and specifies the URL for accessing the target platform. Optionally, user and password for accessing the target can be given, too. See "Proxy Mode" for details.
Example:
--target service:jmx:rmi:///jndi/rmi://bhut:9999/jmxrmi
- --config
-
Specifies a configuration file from where server and check definitions can be obtained. See "CONFIGURATION" for details about the configuration file's format.
Example:
--config /etc/jmx4perl/tomcat.cfg
- --server
-
Specify a symbolic name for a server connection. This name is used to lookup a server in the configuration file specified with
--config
Example:
servers.cfg: <Server tomcat> Url http://localhost:8080/jolokia User roland Password fcn </Server> --config /etc/jmx4perl/servers.cfg --server tomcat
See "CONFIGURATION" for more about server definitions.
- --check
-
The name of the check to use as defined in the configuration file. See "CONFIGURATION" about the syntax for defining checks and multi checks. Additional arguments for parameterized checks should be given as additional arguments on the command line. Please note, that checks specified with
--check
have precedence before checks defined explicitely on the command line.Example:
--config /etc/jmx4perl/tomcat.cfg --check tc_servlet_requests jolokia-agent
- --version
-
Prints out the version of this plugin
- --verbose (-v)
-
Enables verbose output during the check, which is useful for debugging. Don't use it in production, it will confuse Nagios.
- --doc, --help (-h), --usage (-?)
-
--usage
give a short synopsis,--help
prints out a bit longe usage information.--doc
prints out this man page. If an argument is given, it will only print out the relevant sections. The following sections are recognized:- tutorial
-
A 5 minute quickstart
- reference
-
Reference manual explaining the various operational modes.
- options
-
Command line options available for
check_jmx4perl
- config
-
Documentation for the configuration syntax
CONFIGURATION
Using check_jmx4perl
with a configuration file is the most powerful way for defining Nagios checks. A simple configuration file looks like
# Define server connection parameters
<Server tomcat>
Url = http://localhost:8080/jolokia
</Server>
# A simple heap memory check with a critical threshold of
# 90% of the maximal heap memory.
<Check memory_heap>
Value = java.lang:type=Memory/HeapMemoryUsage/used
Base = java.lang:type=Memory/HeapMemoryUsage/max
Unit = B
Label = Heap-Memory: %.2r% used (%.2v %u / %.2b %u)
Name = Heap
Critical = 90
</Check>
A configuration file is provided on the command line with the option --config
. It can be divided into two parts: A section defining server connection parameters and a section defining the checks themselves.
<Server>
With <Server name>
the connection parameters for a specific server is defined. In order to select a server the --server name
command line option has to be used. Within a <Server>
configuration element, the following keys can be used:
- Url
-
The URL under which the jolokia agent can be reached.
- User, Password
-
If authentication is switched on, the user and the credentials can be provided with the User and Password directive, respectively. Currently only Basic Authentication is supported.
- Product
-
The type of application server to monitor. This configuration can speed up checks significantly, but only when aliases are used. By default when using aliases,
check_jmx4perl
uses autodetection for determine the target's platform. This results in at least one additional HTTP-Request. This configuration does not has any effect when MBeans are always used with their full name. - Proxy
-
A HTTP Proxy URL and credentials can be given with the
<Proxy>
sub-section. Example:<Server> .... <Proxy> Url = http://proxy.company.com:8001 User = woody Password = buzz </Proxy> </Server>
- Target
-
With this directive, the JMX-Proxy mode can be switched on. As described in section "Proxy mode",
check_jmx4perl
can operate in an agentless mode, where the agent servlet is deployed only on an intermediated, so called JMX-Proxy server, whereas the target platform only needs to export JMX information in the traditional way (e.g. via JSR-160 export). This mode is especially useful if the agent is not allowed to be installed on the target platform. However, this approach has some drawbacks and some functionality is missing there, so the agent-mode is the recommended way. A sample JMX-Proxy configuration looks like:<Target> Url = service:jmx:rmi:///jndi/rmi://tessin:6666/jmxrmi User = max Password = frisch </Target>
For a discussion about the advantages and disadvantages of the JMX-Proxy mode, please have a look at http://labs.consol.de/ which contains some evaluations of this mode for various application servers (e.g. JBoss and Weblogic).
Single Check
With <Check>
a single check can be defined. It takes any option available also available via the command line. Each check has a name, which can be referenced from the commandline with the option --check name
.
Example:
<Check memory_heap>
Value = java.lang:type=Memory/HeapMemoryUsage/used
Base = java.lang:type=Memory/HeapMemoryUsage/max
Label = Heap-Memory:
Name = Heap
Critical = 90
</Check>
The <Check>
section knows about the following directives:
- Mbean
-
The
ObjectName
of the MBean to monitor. - Attribute
-
Attribute to monitor.
- Operation
-
Operation, whose return value should be monitored. Either
Attribute
orOperation
should be given, but not both. If the operation takes arguments, these need to be given as additional arguments to thecheck_jmx4perl
command line call. In the rare case, you need to call an overloaded operation (i.e. an operation whose name exists multiple times on the same MBean but with different parameter types), the argument types can be given within parentheses:<Check> .... Operation = checkUserCount(java.lang.String,java.lang.String) ... </Check>
- Argument
-
Used for specifying arguments to operation. This directive can be given multiple times for multiple arguments. The order of the directive determine the order of the arguments.
<Check> .... Operation checkUserCount(java.lang.String,java.lang.String) Argument Max Argument Morlock </Check>
- Alias
-
Alias, which must be known to
check_jmx4perl
. Usejmx4perl aliases
to get a list of all known aliases. IfAlias
is given as configuration directive,Operation
and/orAttribute
is ignored. Please note, that usingAlias
withoutProduct
in the server section leads to at least one additional HTTP request. - Path
-
Path to apply to the attribute or operation return value. See "Paths" for more information about paths.
- Value
-
Value is a shortcut for specifying
MBean
,Attribute
andPath
at once. Simply concatenate all three parts via/
(thePath
part is optional). Slashes within MBean names needs to be escaped with a\
(backslash). Example:Value = java.lang:type=Memory/HeapMemoryUsage/used
is equivalent to
MBean = java.lang:type=Memory Attribute = HeapMemoryUsage Path = used
- Base
-
Switches on relative checks. See "Relative Checks" for more information about relative checks. The value specified with this directive defines the base value against which the relative value should be calculated. The format is the same as for
Value
:Base = java.lang:type=Memory/HeapMemoryUsage/max
For relative checks, the
Critical
andWarning
Threshold are interpreted as a value between 0% and 100%. - BaseMBean, BaseAttribute and BasePath
-
As an alternative to specifying a base value in a combined fashion the different parts can be given separately.
BaseMBean
andBaseAttribute
switches on relative checks and specifies the base value. An optionalBasePath
can be used to provide the path within this base value.The example above can be also written as
BaseMBean = java.lang:type=Memory BaseAttribute = HeapMemoryUsage BasePath = max
- Delta
-
Switches on incremental mode as described in section "Incremental Checks". The value given is used for normalization the increase rate. E.g.
Delta = 60
measures the growth rate per minute (60 seconds). If no value is given, the absolute increase between two checks is used.
- Numeric
-
This directive switches on numeric mode, i.e. the given threshold values are compared numerically against the returned JMX value. By default, the check mode is determined by a heuristic algorithm.
- String
-
String checks, which are switched on with this directive, are useful for non-numeric thresholds. See "String checks" for more details.
- Name
-
The name to be used in the performance data. By default, a name is calculated based on the MBean and attribute/operation name.
- MultiCheckPrefix
-
If this check is used within a multi check, this prefix is used to identify this particular check in the output of a multicheck. It can be set to an empty string if no prefix is required. By default the name as configured with
Name
is used. - Label
-
Format for setting the plugin output (not the performance data, use
Name
for this). It takes a printf like format string which is described in detail in "Output Tuning". - PerfData
-
By default, performance data is appended for numeric checks. This can be tuned by setting this directive to "false" (or "0", "no", "off") in which case performance data is omitted. If using this in a base check, an inherited check can switch performance data generation back on with "true" (or "1", "yes", "on")
For relative checks, the value can be set to '%'. In this case, performance data is added as relative values instead of the absolute value measured.
- Unit
-
This specifies how the return value should be interpreted. This value, if given, must conform to the unit returned by the JMX attribute/operation. E.g. for
java.lang:type=Memory/HeapMemoryUsage/used
unit, if set, must beB
since this JMX call returns the used memory measured in bytes. The value given here is only used for shortening the plugin's output automatically. For more details and for what units are available refer to section "Output Tuning". - Critical
-
Specifies the critical threshold. If
String
is set (or the heuristics determines a string check), this should be a string value as described in "String checks". For relative checks, this should be a relative value in ther range [0,100]. Otherwise, it is a simple numeric value which is used as threshold. For numeric checks, the threshhold can be given in the format defined at http://nagiosplug.sourceforge.net/developer-guidelines.html#THRESHOLDFORMAT. - Warning
-
Defines the warning threshold the same way as described for the
Critical
threshold. - Null
-
Replacement value when an attribute is null or an operation returns a null value. This value then can be used in string checks in order to check against null values. By default, this value is "
null
". - Method
-
HTTP Method to use for the check. Available values are
GET
orPOST
for GET or POST HTTP-Requests, respectively. By default a method is determined automatically. The value can be given case insensitively. - Use
-
In order to use parent checks, this directive specifies the parent along with any parameters passed through. For example,
Use = memory_relative_base(80,90),base_label
uses a parent check named
memory_relative_base
, which must be a check defined in the same configuration file (or an imported on). Additionally, the parameters80
and90
are passed to this check (which can be accessed there via the argument placeholders$0
and$1
). See "Parent checks" and "Parameterized checks" for more information about check inheritance.Multiple parents can be given by providing them in a comma separated list.
- Script
-
For complex checks which can not be realized with the configurations described until yet, it is possible to use a Perl script snippet to perfrom arbitrary logic. The content of this script is typically provided as an HERE-document (see example below). It comes with a predefined variable
$j4p
which is an instance of JMX::Jmx4Perl so that it can be used for a flexible access to the server. Note that this scriptlet is executed separately and doesn't not benefit from the optimization done for bulk or relative checks. Check parameters can be accessed as ${0}, ${1}, .. but since these are also valid Perl variables (and hence can be overwritten accidentially), it is recommended to assign them to local variable before using them. In summary, script based checks are powerful but might be expensive.Example:
Script <<EOT my $pools = $j4p->search("java.lang:type=MemoryPool,*"); my @matched_pools; my $pattern = "${0}"; for my $pool (@$pools) { push @matched_pools,$pool if $pool =~ /$pattern/; } return $j4p->get_attribute($matched_pools[0],"Usage","used"); EOT
Includes
Checks can be organized in multiple configuration files. To include another configuration file, the include
directive can be used:
include tomcat.cfg
include threads.cfg
include memory.cfg
If given as relative path, the configuration files are looked up in the same directory as the current configuration file. Absolute paths can be given, too.
Parent checks
With check_jmx4perl
parent checks it is possible to define common base checks, which are usable in various sub-checks. Any <Check>
can be a parent check as soon as it is referenced via a Use
directive from within another check's definition. When a check with a parent check is used, its configuration is merged with this from the parent check with own directives having a higher priority. Parent checks can have parent checks as well (and so on).
For example, consider the following configuration:
<Check grand_parent>
Name grand_parent
Label GrandPa
Critical 10
</Check>
<Check parent_1>
Use grand_parent
Name parent_1
Critical 20
</Check>
<Check parent_2>
Name parent_2
Warning 20
</Check>
<Check check>
Use parent_1,parent_2
Warning 40
</Check>
In this scenario, when check check
is used, it has a Name
"parent_2
" (last parent check in Use
), a Label
"GrandPa" (inherited from grand_parent
via parent_1
), a Critical
20 (inherited from parent_1
) and a Warning
40 (directly give in the check definition).
A parent value of a configuration directive can be refered to with the placeholder $BASE
. For example:
<Check parent>
Name Parent
</Check>
<Check check>
Use parent
Name Child: $BASE
</Check>
This will lead to a Name
"Child: Parent
" since $BASE
is resolved to the parent checks valus of Name
, "Parent"
in this case. The base value is searched upwards in the inheritance hierarchy (parent, grand parent, ...) until a value is found. If nonen is found, an empty string is used for $BASE
.
Parameterized checks
Checks can be parameterized, i.e. they can take arguments which are replaced in the configuration during runtime. Arguments are used in check definition via the positional format $0
, $1
, .... (e.g. $0
is the first argument given). Arguments can either be given on the command line as extra arguments to check_jmx4perl
or within the Use
directive to provide arguments to parent checks.
Example:
<Check parent>
Name $0
Label $1
</Check>
<Check child_check>
Use parent($0,"Check-Label")
....
</Check>
$ check_jmx4perl --check child_check .... "Argument-Name"
OK - Check-Label | 'Argument-Name'= ....
As it can be seen in this example, arguments can be propagated to a parent check. In this case, $0
from the command line (Argument-Name
) is passed through to the parent check which uses it in the Name
directive. $1
from the parent check is replaced with the value "Check-Label
" given in the Use
directive of the child check.
Parameters can have default values. These default values are taken in case an argument is missing (either when declaring the parent check or missing from the command line). Default values are specified with ${
arg-nr:
default}
. For example,
<Check relative_base>
Label = %.2r% used (%.2v %u / %.2b %w)
Critical = ${0:90}
Warning = ${1:80}
</Check>
defines a default value of 90% for the critical threshold and 80% for the warning threshold. If a child check uses this parent definition and only wants to ommit the first parameter (but explicitely specifying the second parameter) it can do so by leaving the first parameter empty:
<Check child>
Use relative_base(,70)
</Check>
Multichecks
Multiple checks can be combined to a single MultiCheck. The advantage of a multi check is, that multiple values can be retrieved from the server side with a single HTTP request. The output is conformant to Nagios 3 multiline format. It will lead to a CRITICAL
value as soon as one check is critical, same for WARNING
. If both, CRITICAL
and WARNING
is triggered by two or more checks, then CRITICAL
take precedence.
If a single check within a multi check fails with an exception (e.g. because an MBean is missing), its state becomes UNKNOWN
. UNKNOWN
is the highest state in so far that it shadows even CRITICAL
(i.e. if a single check is UNKNOWN
the whole multi check is UNKNOWN
, too). This can be changed by providing the command line option --unknown-is-critical
in which case all UNKNOWN
errors are mapped to CRITICAL
.
A multi-check can be defined with the directive <MultiCheck>
, which contain various references to other <Check>
definitions or other multi check definitions.
Example:
<MultiCheck all>
MultiCheck memory
MultiCheck threads
</MultiCheck>
<MultiCheck memory>
Check memory_heap($0,80)
Check memory_pool_base("CMS Perm Gen",90,80)
</MultiCheck>
<MultiCheck threads>
Check thread_inc
Check thread_deadlock
</MultiCheck>
Here a multi check group memory has been defined with reference to two checks, which must exist somewhere else in the configuration file. As it can be seen, parameters can be given through to the check in the usual way (literally or with references to command line arguments). The group all combines the two groups memory and thread, containing effectively four checks.
A multi-check is referenced from the command line like any other check:
$ check_jmx4perl .... --check all 90
(90 is the argument which replaces $0
in the definition above).
The summary label in a multi check can be configured, too.
Example:
<MultiCheck memory>
SummaryOk All %n checks are OK
SummaryFailure %e of %n checks failed [%d]
...
</MultiCheck>
These format specifiers can be used:
%n Number of all checks executed
%e Number of failed checks
%d Details which checks failed
Predefined checks
check_jmx4perl
comes with a collection of predefined configuration for various application servers. The configurations can be found in the directory config within the toplevel distribution directory. The configurations are fairly well documented inline.
common.cfg
Common check definitions, which can be used as parents for own checks. E.g. a check relative_base
can be used as parent for getting a nicely formatted output message.
memory.cfg
Memory checks for heap and non-heap memoy as well as for various memory pools. Particularly interesting here is the so called Perm Gen pool as it holds the java type information which can overflow e.g after multiple redeployments when the old classloader of the webapp can't be cleared up by the garbage collector (someone might still hold a reference to it).
threads.cfg
Checks for threads, i.e. checking for the tread count increase rate. A check for finding out deadlocks (on a JDK 6 VM) is provided, too.
jetty.cfg
Various checks for jetty like checking for running servlets, thread count within the app server, sessions (number and lifing time) or requests per minute.
tomcat.cfg
Mostly the same checks as for jetty, but for tomcat as application server.
websphere.cfg
WebSphere specific checks, which uses the configuration files below the `websphere/` directory. For this checks to work, a customized Jolokia agent with JSR-77 extensions is required. The GitHub project for this enhanced agents can be found at https://github.com/rhuss/jolokia-extra and downloaded at Maven Central (http://central.maven.org/maven2/org/jolokia/extra/jolokia-extra-war/)
LICENSE
This file is part of jmx4perl.
Jmx4perl is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 2 of the License, or (at your option) any later version.
jmx4perl is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with jmx4perl. If not, see <http://www.gnu.org/licenses/>.
AUTHOR
roland@cpan.org