Alert on failed cron jobs

Monitoring cron jobs is essential to ensure scheduled tasks are completed successfully. When a cron job fails, quick notification is critical for timely troubleshooting. Here are several reliable methods to set up alerts for failed cron jobs:

1. Use Cron’s Built-In Mail Alert

By default, cron can email the output of a job to the user who owns the crontab.

Steps:

1. Set the MAILTO variable at the top of your crontab:

   bash
   CopyEdit
   MAILTO="your@email.com"
   

2. Ensure your system has an MTA (Mail Transfer Agent) like sendmail or postfix installed and configured.

If a job produces any output (stdout or stderr), it will be emailed to the specified address.

2. Redirect Output to Email Only on Failure

Send only errors:

bash
CopyEdit
* * * * * /path/to/your/script.sh 2>&1 >/dev/null | mail -s "Cron Job Failed" your@email.com

This command:

• Redirects standard output to /dev/null

• Keeps standard error, which is piped to mail if there’s any error

3. Log and Monitor Exit Codes

Create a wrapper script to check the exit code and send alerts:

bash
CopyEdit
#!/bin/bash

/path/to/your/script.sh
status=$?

if [ $status -ne 0 ]; then
    echo "Cron job failed with exit code $status" | mail -s "Cron Job Failure" your@email.com
fi

Then in crontab:

bash
CopyEdit
* * * * * /path/to/wrapper.sh

4. Use logger to Log to Syslog

Log job failure with logger and monitor logs with tools like Logwatch, Logrotate, or Splunk.

bash
CopyEdit
#!/bin/bash

/path/to/your/script.sh
status=$?

if [ $status -ne 0 ]; then
    logger -p user.err "Cron job failed: /path/to/your/script.sh"
fi

5. Integrate with Monitoring Tools

Examples:

• Nagios/Icinga: Set up a service check for the job.

• Zabbix: Use an external script with trap-based monitoring.

• Datadog: Monitor cron output logs and create alerting rules.

• Prometheus + Alertmanager: Export metrics from scripts (e.g., via exit code logs) and trigger alerts.

6. Slack, Discord, or Webhook Notifications

Integrate alert scripts with APIs to send alerts to chat or incident management platforms.

Example using curl and Slack:

bash
CopyEdit
#!/bin/bash

/path/to/your/script.sh
status=$?

if [ $status -ne 0 ]; then
    curl -X POST -H 'Content-type: application/json' --data 
    '{"text":"Cron job failed on '"$(hostname)"': /path/to/your/script.sh"}' 
    https://hooks.slack.com/services/your/webhook/url
fi

7. Log to a Custom File with Timestamp

Maintain your own error log for analysis or alert processing.

bash
CopyEdit
#!/bin/bash

/path/to/your/script.sh
status=$?

if [ $status -ne 0 ]; then
    echo "$(date): Cron job failed with exit code $status" >> /var/log/cron_error.log
fi

Then monitor /var/log/cron_error.log with any log-watching tool.

8. Use at for Retry on Failure

For non-critical, retryable jobs:

bash
CopyEdit
/path/to/script.sh || echo "/path/to/script.sh" | at now + 5 minutes

This requeues the job via at if it fails.

9. Custom Monitoring with cronwrap or cronitor

• cronwrap: A shell wrapper to log execution and status.

• cronitor.io or healthchecks.io: External services to monitor cron schedules and alert on missed or failed runs.

Example using healthchecks.io:

bash
CopyEdit
* * * * * curl -fsS --retry 3 https://hc-ping.com/your-uuid > /dev/null

Trigger this at the start or end of your script. If no ping is received, healthchecks.io sends an alert.

Best Practices Summary:

Combining local logging, error codes, and external notifications provides robust and timely alerts for cron job failures, minimizing downtime and operational issues.