CronJob

4 min read

Prerequisites: kubernetes

The workload controllers you have met so far keep pods running: a daemonset holds one pod per node alive indefinitely, and if a pod dies it is replaced. Batch work is the opposite shape: run to completion, then stop being scheduled at all. Kubernetes models it in two layers, a Job for “run this until it succeeds” and a CronJob for “create such a Job on a schedule”. Official references: Job, CronJob.

Job: run until it succeeds

A Job creates a pod from a template and watches for the pod to exit 0. If the pod fails, the Job retries with a new pod, backing off exponentially, until it succeeds or exhausts backoffLimit (default 6). Two fields in the template matter more than the rest:

  • restartPolicy must be Never or OnFailure, not the Always that long-running workloads use. Never is the debuggable choice: each retry is a fresh pod, and the failed pods stick around so you can read their logs.
  • backoffLimit bounds the retries. For a job that will get another chance soon anyway (a scheduled publisher, for instance), a low limit like 1 fails fast instead of hammering a broken external dependency.

Completed pods are not deleted by the Job; that is deliberate, the logs are the evidence of what happened. Cleanup belongs to whoever owns the Job.

CronJob: a Job factory on a schedule

A CronJob holds a Job template plus a schedule in standard five-field cron syntax (minute hour day-of-month month day-of-week), and at each tick it creates a new Job. Times are evaluated in UTC unless timeZone is set (timeZone: "Asia/Kathmandu" is valid). Watch one work on devata with a throwaway:

kubectl create cronjob tick --image=busybox:1.36 --schedule='*/1 * * * *' -- /bin/sh -c 'date'
kubectl get jobs -w        # wait two minutes, watch tick-<id> jobs appear and complete
kubectl logs -l job-name --tail=-1 -n default | head
kubectl delete cronjob tick   # owned Jobs and their pods are garbage-collected with it

The fields that shape behavior:

  • concurrencyPolicy: what to do if the previous Job is still running at the next tick. Allow (default) starts another alongside it; Forbid skips the tick; Replace kills the old one. Anything that pushes to a shared destination wants Forbid, because two publishers racing a git push is a mess with no winner.
  • successfulJobsHistoryLimit / failedJobsHistoryLimit: how many finished Jobs to keep around (defaults 3 and 1). Keeping a couple of failures is cheap and preserves the logs that explain an outage after the fact.
  • suspend: true: pause the schedule without deleting anything.
  • One Job can also be fired by hand from the template, which is the standard way to test without waiting for the clock: kubectl create job tick-now --from=cronjob/tick.

Missed ticks, and the sharp edge for a cluster that powers off

A CronJob controller that was not running at tick time (cluster off, controller down) does not fire the missed Jobs retroactively when it returns; missed is missed, and the next tick fires normally. For a cluster like devata that loses power now and then, that alone is fine: a */30 schedule resumes at most 30 minutes after boot.

The sharp edge is the missed-tick counter. When the controller checks a CronJob, it counts how many scheduled times were missed since the last run. If that count exceeds 100 and no startingDeadlineSeconds is set, the controller refuses to schedule the Job at all and logs too many missed start time (> 100), and it stays wedged rather than recovering at the next tick. At a 30-minute cadence, 100 misses is about two days: leave the cluster off over a long weekend and the publisher would come back wedged.

startingDeadlineSeconds is the fix as well as its own feature: it declares how late a Job may start after its scheduled time, and, crucially, it bounds the miss-counting window to that same duration. With startingDeadlineSeconds: 600, the controller only ever counts misses in the last 10 minutes, the count can never reach 100, and the CronJob always recovers on the next tick no matter how long the cluster slept. Any CronJob on a machine that is allowed to be off should set it.