MOTIR-3424Done
(motir-core) Retries, `onFailure` and DLQ parity — the `job_run` ledger and the operator dashboard keep their shape
Makes failure on the new engine behave exactly as it does today, so that the operator surface shipped in MOTIR-64 keeps working without being touched.
What must be preserved, not reinvented
- Named retry policies.
retryPolicy: 'idempotent'and'transient'keep their current attempt counts and backoff. A job that retries five times today retries five times after this card. - The
job_runledger. One row per run:runningat start,succeededon return with the handler's JSON-safe output,failedonce the budget is exhausted. The row must be written exactly once per run even when the handler replays across step boundaries — the existing implementation gets that by writing it inside a memoized step, and the same trick applies here. - The DLQ. On terminal failure a dead-letter row is written and the run is replayable from
/settings/workspace/jobs. onFailuresemantics. The current code moved the dead-letter write OUT of atry/catchinto Inngest'sonFailurehook for a specific reason: on the real executor, astep.runscheduled from a catch block after the terminally-failing step never executes, so the failed/DLQ rows silently never got written in production while the in-process test harness made it look like they did. The Postgres engine must provide an equivalent after-all-retries-exhausted hook, and the reason must be recorded — otherwise the next person simplifies it back into acatchand reintroduces a bug that only appears in production.
Acceptance criteria
- A job exhausting its retries writes a
failedjob_runrow and a DLQ row, asserted against a real Postgres by actually exhausting the budget rather than by calling the hook directly. - Replay from the operator dashboard re-runs the job and succeeds — the existing UI works unchanged against the new engine.
- Exactly one
job_runrow per run, including when the handler replays across steps. - A non-JSON-safe return degrades to a NULL
outputrather than failing the run, as it does today. - The
onFailure-not-catchreasoning is carried in a comment on the new implementation, citing why the catch-block form fails only in production. - Unit and real-Postgres integration tests ship with the change.
Context refs
lib/jobs/defineJob.ts— the ledger writes and theonFailurewiringlib/jobs/dlq.ts·lib/jobs/retries.ts— the DLQ and the named policieslib/services/jobRunsService.ts·app/(authed)/settings/workspace/jobs/page.tsx— the operator surface that must not changedocs/jobs.md§ Run ledger — the contract to preserve