Cron Job Monitoring: Alert on Missed Runs, Not Errors

Q: How do I get alerted when a cron job does not run?

Use a dead man's switch. The job sends a success ping as its last step after the work is committed, and a monitor that knows the schedule and a grace period alerts when that ping does not arrive in time. Error alerts cannot help here, because a job that never starts writes no logs and raises no exceptions for anything to catch.

Q: Why did my Kubernetes CronJob skip a run without any error?

Skipping is documented behaviour. A run is skipped if it cannot start before startingDeadlineSeconds, if concurrencyPolicy is Forbid and the previous run is still going, or if the controller counts more than 100 missed schedules, in which case it logs too many missed start times. None of these produce a failed Job, so only a missed-run alert will catch them.

Q: Which Prometheus metric shows when a CronJob last succeeded?

kube-state-metrics exposes kube_cronjob_status_last_successful_time and kube_cronjob_next_schedule_time. Alert when time() minus the last successful time exceeds one schedule interval plus a grace period, for example 26 hours for a daily job with two hours of grace. Pair it with absent() so the alert still fires if the CronJob object or its series disappears.

Q: What is grace time in cron job monitoring?

Grace time is the extra lateness a monitor tolerates before it alerts. Healthchecks.io defines period as the expected time between pings and grace as the additional wait before a late check alerts; with a start signal, grace also bounds the time between start and success. Set it to realistic slack so ninety seconds of drift does not page anyone.

Q: Should the heartbeat be sent at the start or the end of the job?

Send both, but the one that matters is the success ping at the end, after the work is committed. A start ping only proves the scheduler fired. Start without a matching success tells you the job hung or ran too long, which has a different owner from a run that never started, and the gap between the two gives you duration.

Q: How do I detect a cron job that ran but did nothing?

Check the output, not the job. Query the newest row in the table it writes or the modification time of the file it produces, and alert when the data is older than expected. Add a data-quality assertion such as row counts within expected bounds. Heartbeats alone will show green for a job that exits zero after processing zero rows.

Q: Does concurrencyPolicy Forbid fix overlapping cron jobs?

It prevents overlap by skipping the new run while the old one is still going. That hides a job that has become too slow and converts it into a missed run, which you only see if you alert on missed runs. Fix the duration, make the job idempotent, and monitor maximum run time so a slowing job alerts before it starts to overlap.

Q: When should a scheduled job be replaced instead of monitored?

When it polls for work that is really an event. A job that runs every five minutes to look for new records is usually a missing queue or event subscription, and event-driven consumers fail loudly because consumer lag grows. If the job means react when X happens, build the reaction to X and retire the schedule, along with its monitoring.

Satyam Kumar

返回博客

architecture

A Cron Job Fails by Saying Nothing: Observability for Scheduled Jobs in 2026

By Satyam KumarOctober 31, 202612 min read

cron job monitoring scheduled job observability dead man's switch missed cron job alert kubernetes cronjob monitoring startingdeadlineseconds concurrencypolicy forbid kube-state-metrics cronjob prometheus absent alert heartbeat monitoring healthchecks.io sentry crons data freshness check batch job sla idempotent jobs sre alerting design

A Cron Job Fails by Saying Nothing: Observability for Scheduled Jobs in 2026

Frequently Asked Questions

分享这篇文章

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

人工智能和云架构师。帮助团队构建可扩展到数百万的系统。

LinkedIn GitHub

A Cron Job Fails by Saying Nothing: Observability for Scheduled Jobs in 2026

Frequently Asked Questions

分享这篇文章

Comments

Leave a comment