Motir

Type · deploy (an operator secret write + a ledger read-back; no code change) Parent · MOTIR-3413 (the discovery epic — no blocked_by / blocks edge into a not-done card in the card this was found in, so the edge test says NO) Discovered in · MOTIR-3688, while diffing the census declaration against the live secret to discharge that card's AC 4.

⚠️ AMENDED 2026-08-27T15:06Z — this card also carries ONE observation inherited from MOTIR-3688. See Inherited from MOTIR-3688 below and the criterion that implements it. Nothing else about the card's scope changed; the estimate went 25 → 35 minutes.

The measurement

Taken 2026-08-27T14:25Z, against origin/main at 68eaa0340281665950b41b52746be7a046079ae8 and the live MOTIR_POSTGRES_JOB_IDS read from inside motir-core machine 8576143c4ee538:

census MIGRATED_TO_ENGINE  (tests/jobs/every-job-declares-its-lane.test.ts)  →  38 ids
live   MOTIR_POSTGRES_JOB_IDS                                               →  37 ids

comm -23 census live  →  system.job-run-reap        # declared, NOT routed
comm -13 census live  →  (empty)                    # nothing routed that is undeclared

The census is TOTAL over the registry in both directions (38 migrated + 3 DELIBERATELY_ON_INNGEST = the 41 defineJob ids on origin/main), so the guard test is green and says nothing about this. That is by design and is not the bug — the test's own header explains why it cannot read production. The bug is that nothing else does either.

Root cause / fix

system.job-run-reap (lib/jobs/definitions/jobRunReap.ts, daily 0 6 * * *, catchUp: 'latest') landed in #2352 (MOTIR-3683), merged 2026-08-27T14:19:02Z. The same pull request added its MIGRATED_TO_ENGINE entry — the census working exactly as intended: its author had to name a lane. What no card owns is the SECOND half, which docs/jobs.md states outright: "the census is a DECLARATION; the secret is the DEPLOYMENT — keep them equal by hand". MOTIR-3683 is done and none of its acceptance criteria mention MOTIR_POSTGRES_JOB_IDS.

The consequence is not "no lane" — it is the wrong lane, and it is silent:

• lib/jobs/engine/scheduler.ts:186 — if (!routedToEngine(def.id)) continue; ⇒ the engine gives it no timer.
• lib/jobs/defineJob.ts:483 — if (routedToEngine(id)) return { skipped: 'routed-to-postgres-engine' } ⇒ the Inngest handler does run it, and the deploy's Register the deployed functions with Inngest step registers the function.

So the first release carrying #2352 starts running the reap on Inngest, contradicting the checked-in declaration, and it will lose its subscriber the moment MOTIR-3418 deletes the SDK — the identical failure mode MOTIR-3688 and MOTIR-3682 were filed for, now on its fourth instance in ~30 hours.

Fix direction. After the release carrying #2352 has deployed: read the live value, append system.job-run-reap once, fly secrets set, read it back from inside a machine, and quote the first succeeded engine-lane job_run row. Step 2 is READ-MODIFY-WRITE on a value with no compare-and-swap and two runs have already raced on it — read immediately before writing, write once, read back (docs/jobs.md, the cutover section).

Ordering is not optional. Routing an id whose job is not in the running image routes it nowhere: fly secrets set restarts on the CURRENT release. Deploy first, confirm the job is in the image, then write the secret.

Inherited from MOTIR-3688 — one observation, on the same box

MOTIR-3688 closed 2026-08-27T15:03Z with four of five criteria met. Its AC 2 asked for one succeeded engine-lane job_run row per id: plan-drift/transitioned was quoted (149 rows, all succeeded), and public-follow/digest could not be observed — not a routing question (the id has been in the live secret since 10:44Z and the census declares it) but a release question. Confirmed three ways against the v166 image:

IMAGE=registry.fly.io/motir-core:deployment-01M10E9MC4EWT8YPBSMT4EKVZ8
public-follow/digest      -> ABSENT      # grep -F over /app/worker/worker.mjs
plan-drift/transitioned   -> IN_IMAGE
system.job-run-reap       -> ABSENT

plus relation "public_follow" does not exist in the production database (the release command never ran), and 0 rows for system.public-follow-digest-tick, which arrived in the same #2344 (merged 07:07Z; last deploy v166 at 01:50Z).

It rides here because this card is the one that puts an operator on a post-deploy machine with the ledger already open. The two are independent — neither blocks the other — and this one costs two extra commands.

How to take it, and why it mails nobody. The weekly tick is Monday 09:00 UTC, so use the option MOTIR-3688's body already sanctions — "or by emitting one directly" — with a payload naming a followId that does not exist. publicFollowDigestService.deliverDigest then takes its documented unsubscribed path (if (!follow || !follow.digestOptIn || !follow.confirmedAt) return { sent: false, itemCount: 0 }): the job SUCCEEDS and writes its ledger row while sending no mail and writing no row of its own. Guard it on both sides — assert the probe id is absent from public_follow before inserting, and skip entirely if any public-follow/digest job_run row already exists. Emitting is one job_event row plus one job_queue row (job_id / event_name = public-follow/digest, state='pending', max_attempts=5 for retryPolicy: 'idempotent', event_id in the engine's ^c[a-z0-9]{24}$ shape so the lane discriminator attributes it); the worker's poll claims it without needing a NOTIFY.

Do NOT emit before confirming the job is in the running image — with no subscriber in the registry the row is stranded in production and can land a failed entry against the id being proven. That is the same ordering this card's first criterion already enforces for the secret write.

Resolution

CLOSED 2026-08-27T19:52Z. All six criteria met; the full evidence — every command and its output — is in the run comment on this card. Summary:

• Ordering held. Release v173 (registry.fly.io/motir-core:deployment-01M11W20R8KGN8HBTC3CND5ATX, 15:12Z) was already live and carried #2352; system.job-run-reap and public-follow/digest both grep IN_IMAGE against /app/worker/worker.mjs. Image first, secret second.
• The write. MOTIR_POSTGRES_JOB_IDS went 37 → 38 ids, one append, no duplicates, nothing lost — the value was re-read immediately before the write and byte-compared against the pre-read (no concurrent writer), then read back from inside machine 8576143c4ee538. Both values are quoted verbatim in the comment.
• system.job-run-reap runs on the engine. job_run cmtbxlu3q000ghmpxs56wbihh, lane=engine, status=succeeded, event_id=cmtbxlty6000bhmpxmhit3wi3, started_at=2026-08-27T19:44:31.910Z, for the caught-up 06:00Z fire. Output {"scanned":27,"abandoned":27,"stillLive":0} — it cleared MOTIR-3683's backlog of stranded running rows on its first pass.
• The inherited criterion arrived. public-follow/digest job_run cmtbxrdww00qyhmpxxg84ug0d, lane=engine, status=succeeded, event_id=chwejo6ffrlpdwwabq40nqgcb, started_at=2026-08-27T19:48:50.864Z, output {"sent":false,"itemCount":0} — the unsubscribed path, no mail. MOTIR-3688's AC 2 is now whole and has been told so.
• Set-equal. comm -23 and comm -13 between the census and the live secret both return empty. 38 = 38.
• DELIBERATELY_ON_INNGEST untouched — the three supervisor ids remain ABSENT from the live secret; MOTIR-3489's scope was not taken.

One correction to the inherited half's premise: public_follow DOES exist in production now — the v173 release ran the migration MOTIR-3688 found missing. Both reasons its AC 2 was unobservable were release questions, and one deploy closed both.

The class is still open, and now has an owner. Filed as MOTIR-3716 under this same epic: nothing reconciles MIGRATED_TO_ENGINE against the live secret, so absent that card the next newly-routed job produces a fifth instance of this one, silently. This card's explanation asked for exactly that and placed it out of its own scope; MOTIR-3716 is that scope, relates_to here.

Acceptance criteria

• ORDERING — the release carrying #2352 is deployed BEFORE the secret is written: FLY_IMAGE_REF read from inside a machine differs from deployment-01M10E9MC4EWT8YPBSMT4EKVZ8, and system.job-run-reap is present in the running image (grep -l 'system.job-run-reap' /app/worker/worker.mjs). Both outputs quoted.
• The live MOTIR_POSTGRES_JOB_IDS contains system.job-run-reap, read from inside a machine, with the full value recorded verbatim and no duplicates — and the PRE-write value quoted alongside it, so no id lost in the read-modify-write can hide.
• One engine-lane succeeded job_run row for system.job-run-reap is quoted, with its event_id and started_at.
• INHERITED FROM MOTIR-3688 — one engine-lane succeeded job_run row for public-follow/digest is quoted, with its event_id and started_at, closing that card's unobserved AC-2 half. It is taken by emitting one event directly at a followId asserted absent from public_follow, so no mail is sent — and only after grep -F 'public-follow/digest' /app/worker/worker.mjs confirms the job is in the running image. If the emit is skipped or the row does not arrive, say so on MOTIR-3688 as well as here; a silent omission re-opens exactly the gap this criterion exists to close.
• MIGRATED_TO_ENGINE and the live secret are set-equal at the end: the comm -23 / comm -13 pair is re-run and both sides quoted as empty.
• DELIBERATELY_ON_INNGEST is unchanged — this card does not take MOTIR-3489's scope, and adds no supervisor id to the secret.

Context refs

• lib/jobs/definitions/jobRunReap.ts — the job
• tests/jobs/every-job-declares-its-lane.test.ts — the census (MIGRATED_TO_ENGINE / DELIBERATELY_ON_INNGEST)
• lib/jobs/engine/scheduler.ts:186 · lib/jobs/defineJob.ts:483 — the two reads of the switch that decide the lane
• lib/services/publicFollowDigestService.ts — deliverDigest's unsubscribed path, which is what makes the inherited probe safe
• lib/jobs/engine/dispatcher.ts — what an emit writes (job_event + one job_queue row per routed subscriber)
• docs/jobs.md — the cutover operator section: the declaration-vs-deployment split, the image trap, the read-modify-write race
• MOTIR-3688 · MOTIR-3682 — instances 2 and 3 of this class