Prevent Cron Overlap and Retry Storms

A schedule is unsafe when a new tick can start before the previous run has finished. Overlap can duplicate messages, race state files, exhaust APIs, and make logs look like several unrelated failures.

Design an overlap policy

Choose one policy explicitly:

• skip: if a run is active, record a skipped tick and exit;
• coalesce: remember that work is due, then run once after the active job finishes;
• queue with a hard bound: allow only a small, known backlog; or
• parallelize by independent key: run concurrently only when each item has isolated state and side effects.

For most monitors and publishers, skip or coalesce is safer than unbounded parallel execution.

Add a lease, not a permanent flag

A lock should identify its owner and expire after a carefully chosen lease. A plain running=true file can strand the job forever after a crash. The lease duration must exceed normal run time but still allow recovery from a dead process.

Use a timeout for every network call and for the overall run. Retry only transient failures, use exponential backoff with jitter, and cap both attempts and total elapsed time.

Define terminal outcomes

Every run should end as one of:

• no change;
• completed and verified;
• skipped because another run owns the lease;
• retryable failure exhausted; or
• permanent/manual-review failure.

Do not report success merely because a command started or an API accepted a request.

Overlap-control checklist

• Maximum expected run time is measured.
• A skip, coalesce, or bounded-queue policy is chosen.
• The lock has an owner and expiry.
• Network and whole-run timeouts exist.
• Retries are limited and jittered.
• Side effects use idempotency keys.
• Completion includes read-back or downstream verification.
• Stale-lock recovery is tested.

Common pitfalls

Retrying permanent errors

Authentication, validation, and permission failures usually need configuration or human action. Repeating them quickly only creates noise.

Releasing the lock before verification

The lease should cover the complete critical section, including the state commit and result verification.

Keeping unlimited history

Run history should be bounded or rotated. A monitor that grows one state file forever eventually becomes its own failure mode.

Verification steps

1. Start a deliberately slow test run.
2. Trigger the schedule again while it is active.
3. Confirm the chosen overlap policy occurs exactly once.
4. Terminate the test owner and verify stale-lease recovery.
5. Force a transient error and confirm bounded backoff.
6. Force a permanent error and confirm retries stop.
7. Review history to ensure sensitive payloads were not recorded.

Official reference

Use the official Cron automation guide for current scheduling paths and Script-Only Cron Jobs when no model judgment is needed.