Repository navigation
fix(run-engine): requeue a failed restore at the run's queue position (#5104) - #5114
granbellllc-png wants to merge 1 commit into
Conversation
…triggerdotdev#5104) A run resumed from a checkpoint is re-enqueued at its original queue position, so waiting runs dequeue before new runs. When the restore fails before the run continues (heartbeat stall, supervisor requeue, or dequeue error), tryNackAndRequeue previously nacked the message with no retry time, scoring it at now + backoff. Consequently, the resumed run was placed behind newer runs that were triggered after it. This fix: 1. Adds keepQueuePosition to RunQueue.nackMessage. 2. In tryNackAndRequeue, passes keepQueuePosition: checkpointId !== undefined. 3. On the first requeue of a checkpointed run (message.attempt <= 1), scores the message at its original timestamp instead of applying backoff. Subsequent retries retain exponential backoff to avoid tight retry loops. 4. Adds regression test in restoreOutcome.test.ts verifying that the first requeue of a restore dequeues ahead of newer runs, while subsequent failures apply backoff.
|
|
Caution Review failedThe pull request is closed.
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
|
Hi @granbellllc-png, thanks for your interest in contributing! This project requires that pull request authors are vouched, and you are not in the list of vouched users. This PR will be closed automatically. See https://github.com/triggerdotdev/trigger.dev/blob/main/CONTRIBUTING.md for more details. |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Resolves #5104
Summary
When a run resumes from a checkpoint (e.g. after
triggerAndWait,batchTriggerAndWait, or waitpoints), it is re-enqueued at its original queue position so waiting runs dequeue before newly triggered runs.However, if the restore fails before execution can continue (such as a heartbeat stall, supervisor requeue due to runner crash/OOM, or transient dequeue preparation error),
tryNackAndRequeuepreviously nacked the message with no explicitretryAttimestamp. Under default behavior,RunQueue.#callNackMessagecalculated the message score asnow + backoff. This caused the resumed run to lose its position in the queue, falling behind every run triggered after it—which, in busy queues with low concurrency limits, could delay the run for hours.Solution
RunQueue.nackMessage, addedkeepQueuePosition?: boolean.RunAttemptSystem.tryNackAndRequeue, passedkeepQueuePosition: checkpointId !== undefined.RunQueue.nackMessage, whenkeepQueuePositionis true andmessage.attempt <= 1(the first requeue of a resume),retryAtis scored atmessage.timestamp(its original enqueue timestamp/score) rather thannow + backoff.message.attempt > 1) retain standard exponential backoff, preventing tight retry loops if a restore repeatedly fails. Cold starts (withoutcheckpointId) remain unaffected.restoreOutcome.test.tsverifying that a requeued restore dequeues ahead of runs triggered after it on its first retry, and applies backoff on subsequent failures.