PM2 cron_restart Kills Before It Starts: How My Scheduler Silently Died for Two Days

The Symptom

A content pipeline that publishes every morning (generate → WeChat post → email) went completely silent on 08-28 and 08-29 — two days in a row, with identical log signatures:

• The generate step's output stopped at 07:04:xx. No error, no stack trace. Just nothing after that.
• At 07:05 sharp, two dispatcher startup banner lines appeared in the log.
• For the rest of the day, every dispatcher wake-up printed the same line: "record exists for today (running), skipping."
• At 07:40, the downstream WeChat draft step failed with guard:no-today-queue — there was no article to publish.

No crash logs. No OOM. The dispatcher showed up healthy in the process list. The task had simply evaporated.

The Root Cause

The pipeline's scheduler, pipeline-dispatcher, was at the time a one-shot PM2 process: run one round of checks, execute whatever steps are due, exit. PM2's cron_restart: '*/5 * * * *' woke it up every five minutes.

Here's the trap: when the cron fires and the instance is still running, cron_restart doesn't wait for it, and doesn't skip the round. It kills first, then starts — SIGINT to the running process, its spawned children included, then a fresh instance.

This was invisible in normal operation because a generate round took 42–92 seconds, always finishing well before the next 5-minute boundary. On those two mornings, the candidate pool's 24-hour cap had rolled old entries out of the window around 07:00, quota suddenly freed up, and the inline curation step ballooned (one LLM clustering call took 1m43s, plus fact-checking six briefs). Generate got dragged past 07:05 — and then:

1. At 07:05, PM2 killed the dispatcher along with the generate child that was mid-write.
2. The execution record in the database (PipelineRun) was left stuck in running. Nobody cleaned it up.
3. The idempotency key (step + date) saw "a record already exists for today" and refused to retry for the rest of the day.
4. Every downstream step waited for upstream output that would never come. The whole chain failed silently.

One sentence: putting cron_restart on a "wake every N minutes" scheduler gives every one of its subtasks a hard N-minute timeout — a timeout that raises no error, fires no alert, and leaves behind a zombie running record that blocks the retry path too.

The Fix

Three steps — two fixes plus same-day damage control:

1. Make the dispatcher a resident process. The script gained a --loop mode (an internal tick every 5 minutes, aligned to the wall clock); the PM2 config dropped cron_restart in favor of autorestart: true. Ticks run serially, so a long task merely delays the next tick. It can never be killed by its own schedule.
2. Auto-reap zombie records. Each tick marks any record stuck in running for over 30 minutes as failed, on the assumption the process was killed (usually a deploy or container restart). This check runs before the schedule check, outside the catch-up window — otherwise a zombie discovered after the window would hang as running all day.
3. Same-day rescue: docker exec into the container and run each step's script directly (bypassing the dispatcher), fix the zombie PipelineRun, and the remaining steps resumed on schedule.

The PM2 config now carries a comment to stop anyone from "optimizing" it back:

// ⚠ Do not revert to a cron_restart one-shot process: cron_restart kills the
// running instance before starting a new one. On 08-28/08-29, intel-generate
// ran past 07:05 and was killed together with its child processes;
// PipelineRun stuck in `running` and the whole publish chain was down all day.

Verification: an 8-Minute Local Reproduction

"Kill first, then start" is not prominent in the docs, so it's worth nailing down locally. Minimal demo: a 90-second long task (timestamped log lines for start / heartbeat / done / killed), run under two scheduling shapes side by side (PM2 7.0.4, isolated PM2_HOME):

• Incident group demo-cron-dispatcher: one-shot process that spawns the long task, with cron_restart: '* * * * *' (every minute — the production */5 scaled down)
• Control group demo-loop-dispatcher: resident process, serial ticks spawning the same long task, next tick only after the previous one finishes (the post-fix shape)

Both started together and ran for 8m06s (10:10:48–10:18:54). The incident group's log:

Like clockwork, at every minute boundary: the task dies by SIGINT (the first at 12s — the start happened 12 seconds before a boundary; every later round at 60s/90s), the dispatcher dies with it, a fresh instance starts, repeat. A 90-second task under a 60-second cron never reaches DONE.

The control group over the same window:

The 8-minute tally (raw grep -c output):

	Incident group (cron_restart */1)	Control group (--loop resident)
Task starts (START)	9	4
Completions (DONE)	0	4 (full 90s each)
Kills (KILLED)	9 (8 by cron at minute boundaries + 1 manual stop at teardown)	0
PM2 restart counter ↺	8	0

The control group's tick period was also 60 seconds; the long task simply stretched the effective cadence to roughly one run every 2 minutes. Delayed, not killed — that's the entire difference between the two shapes.

Lessons

1. cron_restart is only for tasks that finish far faster than their period. Its semantics are "guarantee a fresh instance is running at each tick," not "trigger a run at each tick." Leave a multiple of headroom between task duration and period — sized for the worst case (the occasional slow path), not the daily average.
2. Don't let the scheduler share a fate with its workers. The dispatcher dying together with its subtasks is the direct cost of coupling "scheduling" and "execution" in one process tree. With a resident scheduler and serial ticks, a slow task costs you a delay, not a wipeout.
3. Idempotency keys need a zombie-reaping mechanism. "Skip if today's record exists" is correct for preventing duplicate runs — but when the record is stuck in running, the safety latch becomes a seal. Any system using state records for idempotency must answer: what happens when the process writing the record dies midway? A staleness timeout is the bare minimum.
4. When debugging this kind of "skipping" loop, check the running record's startedAt first. A running row that started hours ago with no result since is, in effect, the autopsy report of a killed process.