Motir

Found while reading the ledger during MOTIR-3475. job_run is the surface every wave of this migration was verified against and the table /settings/workspace/jobs renders, so its integrity is load-bearing.

Two defects, one shape

1. Runs enter running and never leave. Measured 2026-08-27, status='running' and older than one hour:

system.daily-health-check         INNGEST  n=24   2026-08-02 .. 2026-08-26
system.code-graph-refresh         INNGEST  n=3    2026-08-20 .. 2026-08-27
system.ci-runner-reap             INNGEST  n=1    2026-08-02
system.ci-runner-provision-sweep  INNGEST  n=1    2026-08-04

Twenty-nine rows. The oldest has been running for 25 days. A daily job has left one behind every single day for three weeks.

2. The failure that ends those runs is written as a SEPARATE row, with an event_id that belongs to no lane.

system.daily-health-check         n=27   last 2026-08-26T09:01:52Z
system.ci-runner-provision-sweep  n=2
system.ci-runner-reap             n=1

These rows' event_id matches neither the engine's cuid (^c[a-z0-9]{24}$) nor Inngest's ULID (^[0-9A-HJKMNP-TV-Z]{26}$), so the lane discriminator that MOTIR-3467 established — and that every read-back in this epic depends on — cannot attribute them at all.

The pairing is exact and daily:

2026-08-26T09:01:16.943Z  INNGEST  system.daily-health-check  running    <- never resolves
2026-08-26T09:01:52.776Z  ?        system.daily-health-check  failed     <- 36s later, different event_id

One logical run; two rows; neither one tells the truth on its own.

Why it matters beyond tidiness

• The operator surface lies in the safe-looking direction. /settings/workspace/jobs shows a job that failed 25 days ago as still running. "Running" reads as healthy-and-working, which is the worst available misreading of a dead job.
• It defeated a check in this very migration. Every wave was read back by grouping job_run on the cuid/ULID lane discriminator. A row matching neither is invisible to that query. It happened not to matter here because the affected runs were pre-existing Inngest ones — but a ?-lane row is a run that no lane audit can see, and lane audits are how this whole epic is being verified.
• It hid MOTIR-3606 in plain sight. A three-week-old daily failure presented as 24 perpetually-running jobs plus 27 unattributable failures. Neither shape says "this has been broken since 2026-08-04".

Not diagnosed — deliberately

I did not establish WHY the second row is written or why the first is never closed. Both plausible stories (a step-level retry record vs. a start row whose completion write is lost on a particular failure path) predict this data, and guessing between them would put a theory on the card instead of evidence. jobRunsService.recordStart / its completion counterpart, and how ledgerIdentity() derives event_id on the failure path, are where to start.

Note the affected runs are all INNGEST/?, none ENGINE — so this may be specific to the Inngest wrapper and may retire itself with MOTIR-3418. Establish that before investing in a fix; if it is Inngest-only, the correct outcome may be to backfill the stale rows and let the retirement remove the cause.

Acceptance criteria

• The mechanism is named with evidence: which code path writes the second row, and which one fails to close the first.
• Whether it can occur on the ENGINE lane is stated either way, with the reasoning — this decides whether the fix is worth building or whether MOTIR-3418 removes it.
• No run can remain running indefinitely: either a completion write that cannot be lost, or a reaper that marks an abandoned run terminal with a distinguishable status. A run stuck in running must not render as healthy.
• Every job_run row's event_id is attributable to a lane, or the ledger exposes the lane explicitly instead of leaving it to be inferred from the id format.
• The 29 existing stale rows are dispositioned — backfilled or explained.

Context refs

• lib/jobs/engine/ledger.ts — ledgerIdentity(), run.eventId ?? run.id
• lib/services/jobRunsService.ts — recordStart and the completion path
• lib/jobs/defineJob.ts — the Inngest wrapper's job-run:start step
• app/(authed)/settings/workspace/jobs — the surface that renders these rows
• MOTIR-3606 — the 23-day-old failure this shape concealed