E2E flake: an in-flight inngest job outlives its test's DB teardown → `recordSuccess` throws + FK-violation cascade crashes the WebServer mid-shard
Symptom (CI, intermittent). A Playwright E2E (bulk-*) shard goes red partway through: from some point on, every remaining test times out with page.goto: net::ERR_ABORTED; maybe frame was detached? (90s each), inflating the job to ~2–3× its normal wall-clock. First observed on PR #1495 bulk-1 (board-scrum.spec.ts:160/183/218 all failed twice), while the SAME specs passed on sibling PR #1492 and on main — so it is not a defect in those specs; it's a shared-server crash they happen to run after.
Root cause (from the WebServer logs). A background inngest job spawned by an earlier test is still in flight when that test's teardown truncates the DB between tests. When the orphaned job then completes:
lib/services/jobRunsService.ts:89recordSuccess()reads thejob_runrow inside the tx and throwsjob_run <id> not found when recording successbecause teardown already truncated it (see also the sibling status-flip paths).- Concurrent job writes that reference now-truncated parents raise a cascade of
DriverAdapterError: ForeignKeyConstraintViolation.
These unhandled rejections degrade the dev WebServer to where subsequent page.gotos abort — so every later test in the shard times out. Because it depends on job/teardown timing, it is non-deterministic and shard-position-dependent; under merge-with-main CI it can red-light unrelated PRs (per CLAUDE.md's flaky-test-is-a-release-blocker rule).
Repro signature (grep the failing job log)
[WebServer] Error: job_run <id> not found when recording successdriverAdapterError: Error [DriverAdapterError]: ForeignKeyConstraintViolation(repeated)Test timeout of 90000ms exceeded→page.goto: net::ERR_ABORTED; maybe frame was detached?
Example: run 28625407163, job Playwright E2E (bulk-1) (84890943046).
Fix direction (pick after a repro)
Two complementary angles; probably want both:
- Harden the job-completion path so a vanished
job_runis a benign no-op, not a throw/unhandled-rejection — e.g.recordSuccess/recordFailuretreat "row gone" as already-terminal and return quietly (an orphaned run from a torn-down test, or a genuinely deleted run, should never crash the process). Guards the server against ANY late-arriving job, not just tests. - Order E2E teardown after in-flight jobs drain (or scope truncation so it can't run under an active job) so tests don't strand jobs mid-flight in the first place — the cleaner long-term fix for the test harness.
Acceptance criteria
- The job-completion path never throws/emits an unhandled rejection when its
job_runrow is absent (unit-tested: callrecordSuccess/recordFailurefor a deleted run → no throw, no server-level rejection). - E2E teardown no longer strands in-flight inngest jobs across a truncation (or the stranded job is provably harmless) — the FK-violation cascade +
ERR_ABORTEDsignature does not reproduce over N repeatedbulk-1runs. - One repo = motir-core.
Notes
- Not caused by PR #1495 (a CI-config-only change: OIDC publish workflow + package.json
repositoryfield). That PR'sbulk-1was re-run and is expected green; this bug is logged as a pre-existing latent flake, not a blocker for it. - Related prior E2E-flake bug (different cause, resolved): MOTIR-989.