<img height="1" width="1" style="display:none;" alt="" src="https://px.ads.linkedin.com/collect/?pid=2877026&amp;fmt=gif">

A Nagios Plugin for Monitoring Spark Streaming Applications

By Prakshal Jain - June 8, 2019


The purpose of building this open-source plugin is to monitor Spark Streaming Applications through Nagios, an Open Source Monitoring tool that we’ve used extensively for Machines, Networks, and Services.

The plugin displays a CRITICAL Alert state when the application is not running and an OK state when it is running properly.

GitHub URL: link

Deployment & Configuration


  • The Nagios Agent should be running on a machine with the `yarn` command available and necessary configurations running on it.

  • If you’re utilizing Kerberos, ensure that the Nagios Agent user has the privileges necessary to obtain an authentication token.

Installation Steps on the Nagios WebServer

  1. Download and save the script responsible for monitoring the spark streaming applications (spark_streaming_monitor.sh) under /usr/local/nagios/etc/libexec/.

  2. Make the script executable:

    chmod +x /usr/local/nagios/etc/libexec/spark_streaming_monitor.sh
  3. Create a service definition for the monitoring service:

    Service Definition
    # Path: /usr/local/nagios/etc/objects/localhost.cfg
    define service{
       use local-service
       host_name localhost
       service_description Spark Monitor <sparkAppName>
       # service_description Spark Monitor TestSpark
       check_command spark_monitor!<sparkAppName>
       # check_command spark_monitor!TestSpark
    # <sparkAppName>: the name of the Spark Streaming Application
    set through the --name flag when submitting the application to spark-

Installation Steps on the Nagios Agent

  1. Create a command definition for the monitoring service:

    Command Definition
    # Path: /usr/local/nagios/etc/objects/commands.cfg
    define command{
       command_name spark_monitor
       command_line $USER1$/spark_streaming_monitor.sh $ARG1$
    # $ARG1$ contains the sparkAppName passed through the service 

Configuration Check

  • Ensure the service is visible on the monitoring dashboard accessible at http://<nagios-hostname>/nagios/.

    Spark Monitor PluginSpark Monitor Plugin


All done! Now, you should be able to monitor any and all of your spark streaming applications at a glance from the Nagios dashboard.

To get the best data engineering solutions for your business, reach out to us at Clairvoyant.

Prakshal Jain

Tags: Data Engineering

Fill in your Details