Skip to content

Retry release 504s in lint-ecosystems installer steps - #2040

Merged
Mikola Lysenko (mikolalysenko) merged 1 commit into
mainfrom
ci-janitor/installer-step-retry
Nov 9, 2026
Merged

Mikola Lysenko (mikolalysenko) merged 1 commit into
mainfrom
ci-janitor/installer-step-retry

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

lint-ecosystems runs on every PR, in the merge queue and nightly, and is part of ci-ok. Three of its steps run scripts/install.sh against real GitHub release assets. The script's curl -fsSL makes one attempt, so a single release-CDN 504 fails the job and ci-ok with it. On 2026-11-09 this happened twice during the same GitHub release-download outage (~05:38–05:47 UTC):

  • PR run 42568357606, job 128999767264 (fix/user-patched-deps-pnpm-yarn-vlt): Downloading SHA256SUMS... curl: (22) ... 504
  • Nightly on main, run 42568829044, job 129001245735: the same 504 on SHA256SUMS

During that outage the same 504s also failed the vexctl download (#2039 fixes that) and the ruby/setup-ruby downloads in two e2e legs. In the merge queue, each of these failures would evict an entry.

Root cause

The test reaches the live network on purpose, because it checks the real latest release end to end. But the download has no tolerance for transient errors.

Fix

A new step writes $RUNNER_TEMP/curl-retry/.curlrc containing retry = 7. The three sh scripts/install.sh invocations run with CURL_HOME pointing there, and curl reads its config from that directory. That gives the installer's curl a retry with exponential backoff (~2 min in all) on transient errors only: timeouts and HTTP 408/429/5xx. A 404, a missing checksum entry or a checksum mismatch still fails immediately. The probe's curl -sfI -m 20 in the Socket-origin step is not affected.

scripts/install.sh is not changed: install.socket.dev must serve a byte-for-byte copy of it (installer-drift.yml), so changing it needs a depscan redeploy. Adding --retry to the user-facing installer may be worth doing separately, as a coordinated change.

Proof

  • Local repro: a server that answers 504 three times and then 200. Plain curl -fsSL exits 22. With CURL_HOME pointing at the retry = 7 .curlrc, curl retries and exits 0 with the body (curl 8.5.0; ubuntu-24.04 runners ship the same 8.5.x).

  • The full CURL_HOME=... sh scripts/install.sh installs the latest release locally.

  • actionlint: same 175 findings before and after, none new. zizmor --offline: 54 findings before and after. YAML parses. python3 -m unittest discover -s scripts/tests: 411 tests OK.

  • On this PR's CI, during the ongoing release-download degradation: attempt 1 (job 129039901803) got 504s on all 8 tries over ~2.3 min and failed. Attempt 2 (job 129042262806) got 504s on 6 tries (07:51:39 → 07:52:10), then installed on the 7th and passed. All three installer steps succeeded. Without this change, that run would have failed on the first 504 as well.

Tests moved or removed

None. Every step runs where it ran before. Job and check names are unchanged.

https://claude.ai/code/session_011f9aeSwgKAzXeLTNSLn5pg


Generated by Claude Code

The three installer steps in lint-ecosystems run scripts/install.sh
against real GitHub release assets, and its single-attempt
`curl -fsSL` turned every release-CDN 504 into a red lint-ecosystems
and so a red ci-ok: twice on 2026-11-09 (a PR run and the nightly).

Write a .curlrc with `retry = 7` under RUNNER_TEMP and point CURL_HOME
at it only for the install.sh invocations. Transient errors (timeouts,
408/429/5xx) retry with exponential backoff, ~2 min in all; a 404 or
checksum mismatch still fails at once. install.sh is left untouched
because install.socket.dev must serve a byte-for-byte copy of it.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011f9aeSwgKAzXeLTNSLn5pg
@mikolalysenko Mikola Lysenko (mikolalysenko) added the ci-janitor Opened by the CI janitor routine (flakes, redundant tests, CI perf) label Nov 9, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

bugbot run


Generated by Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit e6775d9. Configure here.

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

lint-ecosystems (and so ci-ok) failed on e6775d9. The cause is ongoing GitHub degradation, not this diff, but it is the failure this PR targets: job 129039901803.

  • The new retry fired as intended: 8 attempts at 1/2/4/8/16/32/64 s backoff (07:44:33 → 07:46:52), each a 504 on releases/download/v5.0.1/socket-patch-x86_64-unknown-linux-musl.tar.gz.
  • I reproduced it from outside CI at the same time. That URL takes ~11 s per request and returns 504 on about 1 in 3 requests (302 otherwise). SHA256SUMS from the same release returns 200. The same endpoint has been returning 504s since ~05:38 UTC; that is also why the nightly failed.
  • Every other job on this head passed. I re-ran the failed jobs once.

The retry turns short 504 bursts into a pass. It can't outlast a release-download outage that lasts several minutes, and a longer window would run into the job's 10-minute timeout across three installer steps. If this degradation keeps up, the more durable follow-up is to have only the first installer step download the release and point the alternate-origin step at a local mirror. That changes what the step tests, so it needs an owner's call and isn't part of this PR.


Generated by Claude Code

@mikolalysenko Mikola Lysenko (mikolalysenko) added the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label Nov 9, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Ready for review at e6775d93381628c2149eeb257da0fe6c06efa987.

  • CI: 32/32 checks pass (including ci-ok and lint-ecosystems on the re-run of the 504-hit job), 19 skipped. No main-wide failures.
  • Bugbot reviewed e6775d9 and found no issues.
  • For the reviewer: the retry rides out short release-CDN 504 bursts but not a multi-minute outage (see the previous comment for the follow-up idea, which needs an owner's call).

Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Final review brief

What it does. The three lint-ecosystems steps that run scripts/install.sh against real release assets now set CURL_HOME to a temp dir whose .curlrc says retry = 7. A release-CDN 504 then gets retried with backoff (~2 min) and no longer fails ci-ok. install.sh itself is unchanged, so the hosted copy at install.socket.dev doesn't drift.

Risk: low. This is a CI-only change in one job. curl's --retry covers only transient errors (timeouts, 408/429/5xx), so a 404 or a checksum mismatch still fails at once. install.sh calls plain curl -fsSL without -q, so it does read the .curlrc.

Look here

Verified

  • Read the full diff. It is 1 file, +17/−1, and CHANGELOG.md is untouched.
  • Confirmed that install.sh downloads with curl -fsSL (no -q/--disable), so CURL_HOME takes effect. The wget fallback isn't used on these runners.
  • The job is Linux-only (depot-ubuntu-24.04-4 / ubuntu-latest), so there's no Windows path concern.
  • One edge case, not a blocker: if every download in all three steps used up its retries, the job would reach its 10-minute timeout. The result is the same failure, just slower.
  • CI on e6775d9: 32 passed, 19 path-skipped, ci-ok and clippy green. The PR body links an attempt that installed on the 7th retry during the 504 outage.
  • Cursor Bugbot: success, no comments. No review threads.

Changes I made. None.

Auto-merge is armed, so approving sends this straight to the merge queue.


Generated by Claude Code

@mikolalysenko
Mikola Lysenko (mikolalysenko) added this pull request to the merge queue Nov 9, 2026
Merged via the queue into main with commit 5f84319 Nov 9, 2026
93 of 95 checks passed
@mikolalysenko
Mikola Lysenko (mikolalysenko) deleted the ci-janitor/installer-step-retry branch November 9, 2026 19:50
Mikola Lysenko (mikolalysenko) pushed a commit that referenced this pull request Nov 9, 2026
Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-janitor Opened by the CI janitor routine (flakes, redundant tests, CI perf) Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants