bug: a resumed run whose restore fails is requeued behind newer runs #5104

Provide environment information

  System:
    OS: Linux 6.8 Ubuntu 24.04.5 LTS 24.04.5 LTS (Noble Numbat)
    CPU: (16) x64 AMD Ryzen 7 7700 8-Core Processor
    Memory: 48.87 GB / 61.89 GB
    Container: Yes
    Shell: 5.2.21 - /bin/bash
  Binaries:
    Node: 24.22.1 - /usr/local/bin/node
    Yarn: 1.22.22 - /usr/local/bin/yarn
    npm: 11.19.1 - /usr/local/bin/npm
    pnpm: 12.15.0 - /usr/local/bin/pnpm

The runs were on Trigger.dev Cloud: prod environment, us-east-1. Tasks were deployed with the trigger.dev CLI and @trigger.dev/sdk 4.5.14, runtime: "node-24". The output above is from the machine we reproduced on.

Describe the bug

A run that resumes after a wait is enqueued at its original queue position. continueRunIfUnblocked says so: "Re-enqueue with the original timestamp, so waiting runs dequeue before new runs." If the restore then fails before the run continues, the run is requeued through tryNackAndRequeue with no retry time. RunQueue then scores the message at now + backoff (internal-packages/run-engine/src/run-queue/index.ts, #callNackMessage). The run loses its position and waits behind every run that was triggered after it.

What we saw on 2026-11-06:

• Two parent runs called batchTriggerAndWait with about 50 children each, and were checkpointed while they waited. Their queue has concurrencyLimit: 4 and held a backlog of about 100 queued runs.
• All children ended by 21:03 and 21:59 UTC. A few children were OOM-killed, and each of those was marked CRASHED (TASK_PROCESS_OOM_KILLED) 300 s after it started.
• Each parent was then requeued once, at 22:21 and 22:04 UTC. After that the API showed status: "QUEUED", attempts: [], and a trace that ends at batchTriggerAndWait(). In the dashboard the run looks as if it never started.
• The parents stayed queued for over 70 minutes while runs created after them kept starting in the same queue. We cancelled them at 23:32 UTC. That was about 4.5 minutes before the last run that was queued ahead of them started, so we think they were waiting their turn, not lost.
• Each requeue freed a queue slot that another run took within 0.2 s. As far as we can tell from the code, only tryNackAndRequeue writes PENDING for a run that is resuming from a checkpoint.

Expected: a resume whose restore has to be retried keeps its place ahead of runs triggered after it, as the first resume does.

We can't see why the restores failed. The parents ran on a small machine and were close to its memory limit when they started waiting, so we think the restored process was OOM-killed. That part is our sizing. See "Additional information" for how such a restore is reported.

Reproduction repo

https://github.com/PlunderStruck/trigger.dev/tree/repro/resumed-run-queue-position

To reproduce

The branch adds one run-engine test file, explained in repro/resumed-run-queue-position/README.md. It uses containerTest and a real RunEngine. The tests assert today's behaviour, so they pass on main:

cd internal-packages/run-engine
pnpm run test ./src/engine/tests/resumedRunQueuePositionRepro.test.ts --run

Each test:

1. Triggers a parent in a queue with concurrency limit 1, starts it, and blocks it on a waitpoint.
2. Checkpoints it, then completes the waitpoint, so the parent is enqueued for a restore and dequeued ahead of newer runs, as designed.
3. Triggers a newer run while the restore holds the slot.
4. Fails the restore before continueRunExecution, in one of two ways:
   • Stall. The restored runner never continues, so the PENDING_EXECUTING heartbeat requeues it.
   • Supervisor report. The supervisor reports the restore as requeue, as classifyRestoreFailure does for a Runner that failed with reason OOMKilled.
5. Asserts that the parent is PENDING with its checkpoint, that the next dequeue returns the newer run, and that the parent is dequeued only after the newer run finishes.

Suggested fix

These are ideas from reading the code; you know the trade-offs far better than we do.

• Keep the position on the first requeue of a resume. tryNackAndRequeue already knows whether the run has a checkpoint. It could ask nackMessage to redeliver at the message's original score, as the resume enqueue does. Later requeues would keep the backoff, so a restore that keeps failing does not spend the 12 queue attempts in a tight loop. Cold starts would be unchanged. We put this on a branch with a regression test in restoreOutcome.test.ts, which fails without the change: main...PlunderStruck:trigger.dev:fix/resumed-run-queue-position. It adds 18 lines to runAttemptSystem.ts and run-queue/index.ts, most of them comments. We'd be glad to open it as a PR if that's welcome.
• Not covered by that branch: the direct runQueue.nackMessage calls in dequeueSystem.ts and engine/index.ts (error and admin paths) still requeue a resume at now + backoff.

Additional information

• An OOM-killed restore is retried on the same machine with no reason recorded. classifyRestoreFailure (apps/supervisor/src/workloadManager/runCrd.ts) maps OOMKilled to its default { outcome: "requeue" }. The runner-kill reporter also skips a run whose snapshot is not EXECUTING. The run therefore records no attempt and no error, retry.outOfMemory can't move it to a larger machine, and each retry goes to the back of the queue until the 12 queue attempts are spent. Mapping OOMKilled to a failure with TASK_PROCESS_OOM_KILLED would let retry.outOfMemory apply. We haven't changed this, because we can't confirm the restore reason on Cloud.
• No trace or attempt event for a requeue. tryNackAndRequeue emits no run event. A run that has resumed and been requeued looks the same as one that has never started.
• Related but different: batchTriggerAndWait parent never resumes after all child runs complete (batch stuck in PROCESSING with runs: []) #4971 and batchTriggerAndWait parent never resumes after its batch is COMPLETED (follow-up to #4971) #4988 (the parent was never resumed) and bug: run stays EXECUTING after its task finished when the attempt completion is not recorded #5060 (the run stayed EXECUTING after its task finished). Here the parent is resumed but then requeued behind newer runs.
• We can share the run IDs privately.