Skip to content

Stop fetching patch views during hosted scans - #2021

Merged
Mikola Lysenko (mikolalysenko) merged 10 commits into
mainfrom
refactor/hosted-scan-no-patch-view
Nov 9, 2026
Merged

Mikola Lysenko (mikolalysenko) merged 10 commits into
mainfrom
refactor/hosted-scan-no-patch-view

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

LLM Description written by Claude Code:claude-opus-5-5

Summary

Hosted scans no longer fetch /patches/view on the normal path. A view carries every patched file's blob, and v5 keeps no hosted ledger, so the scan fetched full patch bodies only to feed checks within the same run. Each check now uses data the scan already has, or fetches narrowly:

Consumer Now
In-run --vex fetches the records it attests via vex_sources::fetch_records, like standalone vex
Gem stale-install probe sha256 of <home>/cache/<leaf>.gem (and committed vendor/cache) vs the grant's artifact sha256; fallback: that patch's view record
Python stale-install probe the pinned patched wheel's RECORD (downloaded, sha256-checked, only for an installed copy with matching WHEEL tags); fallback: that patch's view record
vlt warm-tree heal the hidden lock's integrity; no readable hidden lock means undeterminable, which already fails closed
In-memory engine / Node binding no stale probe, no view; fetchPatch is accepted and ignored

An agent-mode manifest record of the same patch takes precedence in both probes.

Owner decision (review thread on hosted.rs:405)

When the gem archive or wheel evidence is missing, can't be fetched, or can't judge the copy (pruned cache/, gem cleanup, no patched sha, sdist pin, failed download or sha check, another build's install), the probe fetches that purl's view record and judges the copy from its afterHash map, at most once per uuid and only if no other variant already proved the copy patched. If the view fetch also fails, or under --offline, the copy is unjudged: it leaves assume_applied and goes to VexBuildParams::unjudged, which in-run --vex withholds as stale_install_unverified, even with --vex-no-verify. It is never treated as not stale.

Behavior changes

  • record_fetch_failed is gone; an in-run VEX view failure is vex_record_fetch_failed, and an all-failed run ends with a record_unavailable skip.
  • New VEX omission reason stale_install_unverified (see above).
  • The Python probe may download the pinned patched wheel (once per patch, only with an installed copy).
  • vlt: a FIFO/symlinked hidden lock no longer invalidates the store entry; the advisory asks for vlt ci.

Where to look

  • scan/hosted.rs: ViewFallback, gem_stale_install_warnings, the assume_applied / unjudged wiring
  • scan/hosted/python.rs: Evidence::judge and the pending → view pass
  • vex.rs: the unjudged block after known_stale
  • core/hosted/wheel_record.rs: judge_installed_wheel returns None when it can't judge
  • CLI_CONTRACT.md (gem/Python stale-install guards, omission table), docs/testing/poetry-compatibility.md

Testing

  • New unit tests (gem, Python, VEX): missing archive / wheel 500 / sdist → one view fetch, judged from the record; view 500 or --offline → unjudged (no requests offline); archive or wheel that judges → zero view requests; unjudged purls withheld even when assumed applied and under --no-verify.
  • New e2e: pruned gem cache judged by the view; unjudgeable gem not attested by --vex; patched gem install makes zero view requests; pdm wheel 500 → view fallback.
  • Ran locally: cargo fmt --check, cargo clippy --workspace --all-features -- -D warnings, CLI lib, core vlt_heal / wheel_record / hosted::, e2e_redirect_gem_stale_install, in_process_redirect, pdm/pipenv/poetry redirect suites, hosted_memory_engine, hosted_memory_parity, scan, e2e_redirect_vlt_build.
Review history

🤖 Generated with Claude Code

A hosted scan of a vlt project downloaded every granted patch archive
the way vlt fetches it, and withheld the dependency
(`redirect_vlt_artifact_unverifiable`) when the server re-gzipped it.
That guarded against patch.socket.dev's CDN re-encoding archives,
which vlt rejects with EINTEGRITY. The patch server has sent archives
with `Cache-Control: no-transform` since depscan #26856 and #26867,
and production serves them unencoded today. On any mirror that still
re-encodes, `vlt ci` fails closed with EINTEGRITY, so nothing installs
unpatched bytes.

So the scan no longer fetches the archives. It pins vlt dependencies
directly, without a full download of every patched package per run.
The in-memory hosted engine (the Node addon and `hosted-bundle`) has
no network, so it used to withhold every vlt dependency as "offline";
it now pins them the way the disk flow does.

The warm-tree heal now checks the Socket-owned lock instances this run
confirmed in `vlt-lock.json`. With no patch record and no hidden lock
it reports the copy as undeterminable (asking for `vlt ci`) instead of
comparing it against downloaded archive bytes.

The vlt backtest still probes the live artifact itself: a must-patch
hosted cell records `blocked-by-server-encoding` when its probe sees a
content-encoded response, without running the CLI.

Assisted-by: Claude Code:claude-opus-5-5
A vendored run decides before any download which patches the engine
would refuse, so it never fetches a view, asks for a grant or strips a
hosted pin for a package it cannot vendor. That decision was assembled
in the CLI from five sources (the Bun and vlt preflights, the
package-lock project refusal, the lock-text gates and the pnpm
takeover), and each entry point merged its own subset by hand. They
had drifted: the download phase never asked the package-lock project
refusal or the pnpm takeover gates, `get <uuid>` raised only the Bun
and vlt refusals before the engine, and the vendor loop judged only
Bun and vlt before its takeover.

All of it now lives in one core function,
`socket_patch_core::vendor::refusals::vendor_refusals`, which runs the
checks in the engine's order and scopes the lock-text gates (hosted
pins, the ledger's idempotency skip, packages that never reach their
backend). The scan preview and rollout plan, the download phase, both
`get` paths and the vendor loop call it. The CLI's bun_preflight.rs
and vlt_preflight.rs are gone, and their tests moved to core.

What users see: a v1 package-lock project, or a hosted pnpm pin whose
takeover the pnpm gates refuse, is now refused before its patch view
is fetched; `get <uuid> --mode vendored` raises every such refusal
with its pre-record error envelope and writes nothing; and `vendor`
refuses them before restoring a hosted pin instead of restoring and
rolling back. Codes and details are unchanged.

Assisted-by: Claude Code:claude-opus-5-5
A hosted scan fetched the full patch view (every patched file's blob
content) for each confirmed redirect, only to judge installed copies
and feed the same run's VEX. It now makes no view call at all:

- In-run `--vex` fetches the records it attests from itself, through
  the same record path the standalone `vex` uses, so the scan pipeline
  never asks for them.
- The gem stale-install probe judges an installed gem by the archive
  RubyGems installed it from (`<home>/cache/<leaf>.gem`) against the
  patched artifact's sha256.
- The Python stale-install probe downloads the patched wheel the lock
  now pins, through its grant URL and verified against the pinned
  sha256, and compares the installed copy with the wheel's own RECORD
  when the installed WHEEL tags match. It downloads only when a copy
  is installed.
- Both probes still treat an agent-mode manifest record of the same
  patch as proof of an in-place patch.
- The vlt warm-tree heal judges a store entry by vlt's hidden lock
  alone: the pinned integrity there is healthy, another is stale, and
  without a readable hidden lock the copy is kept and reported as
  unchecked.
- The in-memory hosted engine no longer calls the provider's
  `fetchPatch` or warns `record_fetch_failed`. The Node binding still
  accepts a `fetchPatch` provider method and ignores it, so existing
  providers keep working.

Assisted-by: Claude Code:claude-opus-5-5

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

…c) e2e

judge_installed_wheel read the installed WHEEL and every RECORD path with
bare std::fs reads. Site-packages is user-writable, so a FIFO there would
block the hosted scan's blocking pool. Read through
read_regular_*_sync so a FIFO or device counts as unreadable, and add a
regression test for both the WHEEL and a RECORD path.

The real-vlt e2e for heal rule (c) without a hidden lock still expected
the old record-based heal (copies invalidated). Hosted scans no longer
fetch patch records, so that case is now undeterminable: the copies are
kept and the advisory says to run `vlt ci`. Match the in-process test and
check that `vlt ci` then installs the patched package.

Co-Authored-By: Claude <noreply@anthropic.com>
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

bugbot run


Generated by Claude Code

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Labeled Ready for review by the burn-down agent.

  • Head: fb3d4183852150c00139b671240b93019ba88655
  • CI: 49/49 check runs green (30 success, 19 skipped by path filters)
  • Fixed in fb3d418:
    • e2e_redirect_vlt_build rule (c) without a hidden lock still expected the old record-based heal. This PR makes that case undeterminable (copies kept, advisory says run vlt ci), so the e2e now matches the in-process test and checks that vlt ci installs the patched package.
    • Bugbot (medium): the wheel probe read site-packages files with bare std::fs reads, so a FIFO could block the scan. It now uses the regular-file guard, with a FIFO regression test.
  • Bugbot: re-reviewed fb3d418, no new issues.
  • Reviewer note: stacked on Refuse vendored patches the same way everywhere #2015, which is stacked on Stop downloading vlt artifacts during hosted scans #2013. heal_rule_c_no_record now covers the same scenario as heal_rule_c_no_hidden_lock because patch records are never fetched; consider dropping one of them.
  • Slack announcement not sent this run (Slack send tool unavailable); next run retries.

Generated by Claude Code

@mikolalysenko Mikola Lysenko (mikolalysenko) added the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label Nov 4, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Final review brief (for the human approver)

What it does: hosted scans stop calling /patches/view, which returned the full patched file contents just to feed checks inside the same scan. In-run --vex now fetches its own records, the same way standalone vex does. The gem stale-install probe compares the sha256 of the cached .gem with the grant's. The Python probe downloads the patched wheel, verifies its sha256 and reads its RECORD. The vlt heal trusts only vlt's hidden lock. Stacked on #2015, so it merges into that branch, not main.

Risk: medium. I found no security problem and nothing that makes the probes warn wrongly. The risk is in the Python probe: it now judges an installed copy only when its WHEEL tags and dist-info name exactly match the first patched wheel for that patch. Where they don't, a stale venv goes unwarned with no signal (see the open questions).

Look here

  • wheel_record.rs:40-50: checks the sha256 before it parses anything. Downloads are capped at 256MB and RECORD/WHEEL reads at 16MB. Nothing is extracted. RECORD paths go through is_safe_relative_subpath.
  • wheel_record.rs:134-171:`` the tag gate.
  • python.rs:91-110:``` how the override is chosen and the --offline skip.
  • hosted.rs:387-407:```` any cached .gem whose sha isn't a patched one now warns as stale.
  • vlt_heal.rs:258-286:````` with no readable hidden lock the result is now Undeterminable (it used to be `Healthy`), and the in-run VEX drops it. That errs on the safe side.

Verified: I read the PR's own diff (42843ef..fb3d4183852150c00139b671240b93019ba88655, 31 files). No view calls remain on the scan, memory-engine or Node paths. --offline and --dry-run are honoured. CRLF in RECORD is handled. CHANGELOG.md is untouched and there is no debug leftover. Ran locally: wheel_record unit tests (5), CLI stale (23), in_process_redirect_poetry and e2e_redirect_gem_stale_install, all passing. The poetry test depends on the served wheel, so it exercises the new probe. CI: 30 passed and 19 skipped by path filter (49 checks). Bugbot: passing, and its one thread is resolved.

Changes I made: none.

Open questions / nits (none block merging)

  1. When a patch ships several platform wheels, or the patched wheel changes its dist-info name or tags (PyYAML vs pyyaml, py2.py3 → py3), the probe never judges the installed copy. Try every matching .whl override? This could be a follow-up.
  2. Every re-scan downloads the whole patched wheel again for each pypi candidate that has an installed copy and no local record. That costs about what the old view fetch did.
  3. No test covers the --offline skip of the wheel probe, or a gem with no .gem under <home>/cache (git or path sources).
  4. Nit: hosted.rs:974-976 still says "patch records (fetched below)", but that fetch is gone.

Auto-merge is not armed. This PR targets #2015's branch, which has no review rule, so arming it would merge it immediately. Approve and merge it after #2015, or retarget it to main once #2013 and #2015 land.


Generated by Claude Code

Base automatically changed from refactor/vendor-refusals-into-core to main November 6, 2026 22:13
@mikolalysenko Mikola Lysenko (mikolalysenko) removed the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label Nov 6, 2026
Main squash-merged this branch's stacked base commits as #2013 and #2015
with review changes, so the merge conflicted in the vlt heal code and
tests. Resolved by taking main and re-applying only this PR's own two
commits on top, which apply cleanly.

Co-Authored-By: Claude <noreply@anthropic.com>
The record-fetch-failure test that mounted a 404 mock is gone, so
Mock, method and path were left unused after merging main.

Co-Authored-By: Claude <noreply@anthropic.com>
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

bugbot run


Generated by Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit 4298af6. Configure here.

@mikolalysenko Mikola Lysenko (mikolalysenko) added the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label Nov 6, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Re-labeled Ready for review by the burn-down agent after resolving merge conflicts.

  • Head: 4298af6
  • Conflicts: main squash-merged this branch's stacked base commits as Stop downloading vlt artifacts during hosted scans #2013 and Refuse vendored patches the same way everywhere #2015, so the merge conflicted in the vlt heal code and tests. The merge (1c97b7f) takes main and re-applies only this PR's own two commits (9bcb4c5, fb3d418), which apply cleanly on top. A follow-up commit drops the Mock/method/path wiremock imports that became unused once the record-fetch-failure test went away.
  • CI: 50/50 check runs settled green (31 success, 19 skipped)
  • Bugbot: reviewed 4298af6, no findings
  • Locally: cargo test -p socket-patch-core --lib and the in_process_redirect / hosted_memory_engine tests pass, except 7 chmod-based write-failure tests that cannot fail as root in the agent container (CI runs them green).
  • Reviewer note: the diff against main is now only the patch-view removal and the wheel probe change. The vlt rule (c) heal now reports "undeterminable" when there is no hidden lock, rather than using a record.
  • Slack announcement could not be sent from this run (no Slack send tool available).

Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Final reviewer: blocked on an open review thread, needs an owner decision (agent:needs-human). Head 4298af60.

The review thread on hosted.rs:405 is unresolved and has no reply. I checked it against the head: when there is no agent-mode record and no <home>/cache/<leaf>.gem, the gem probe (hosted.rs:395-405) returns InstalledRecordJudgment::default(), which means "not stale". On main the fetched patch view's afterHash judged that case. The thread says the PyPI path has the same gap (an sdist pin, or a failed wheel download or sha256 check). It also says in-run VEX then attests the purl through assume_applied.

Question for the owner: when the archive or wheel evidence is missing or can't be fetched, which should the scan do?

  1. Fall back to fetching the patch-view record for that purl only. This keeps the old safety, but it puts back some of the fetches this PR removes.
  2. Treat the purl as unjudgeable. Add it to known_stale or drop it from assume_applied, so VEX doesn't attest it. No fetch, but it may give false "stale" warnings on images that prune caches.

CI is green (31/31 non-skipped) and Bugbot is clean on this head, so this thread is the only thing blocking it. I haven't posted a review brief.


Generated by Claude Code

@mikolalysenko Mikola Lysenko (mikolalysenko) removed the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label Nov 9, 2026
Resolve vlt_heal.rs imports: main's #2002 moved hosted_patch_uuid to
crate::origin; this PR dropped the patch-view helpers, so keep only the
origin import. Main's new go-workspace vex_sources tests build Sources
without the hosted_uuids field this PR adds; give them the empty list
every other test uses.

Co-Authored-By: Claude <noreply@anthropic.com>
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Burn-down agent: merged main (568e376) to clear the conflict in the vlt_heal.rs imports. Kept main's crate::origin::hosted_patch_uuid and dropped the patch-view imports this PR removes. Main's two new go-workspace tests in vex_sources.rs now also pass hosted_uuids: Vec::new(). New head 3130bbd; vex_sources tests pass locally. Removed Ready for review until CI is green; the agent:needs-human question is still open.


Generated by Claude Code

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[agent] Review findings (Claude Opus + codex, independent passes) — same defect as the open thread at hosted.rs:405:

  1. Unjudgeable gem is treated as not stale (crates/socket-patch-cli/src/commands/scan/hosted.rs:387-405, both reviewers). With no agent-mode record and no readable <home>/cache/<leaf>.gem, the probe returns InstalledRecordJudgment::default(), the purl stays in assume_applied, and VEX (which now fetches the record itself) attests it. Input: image built with bundle install && rm -rf $BUNDLE_PATH/cache → scan --mode hosted --vex attests the unpatched installed gem. Main compared against the view record's afterHash and flagged it stale.
  2. Same gap for PyPI (scan/hosted/python.rs:46,90-108, crates/socket-patch-core/src/hosted/wheel_record.rs:141-144). An sdist pin, a missing integrity.sha256, a failed wheel download/sha mismatch, a WHEEL-tag mismatch, or (codex) a legacy *.egg-info install without <dist-info>/WHEEL all fall to default → no redirect_pypi_stale_install, not in known_stale, attested.

Suggested fix: judge with the record VEX already fetches, or put any purl the probe can't judge into known_stale / drop it from assume_applied.

Minor: deleting rule_c_bytes_mismatch_whatever_the_hidden_lock_says (vlt_heal) may open a similar path if vlt can rewrite its hidden lock without re-extracting; the probe uses only the first .whl override per uuid.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[agent] Approving per maintainer decision; see the findings in the review above.

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Final reviewer (head 3130bbd): approved and CI green. The unresolved thread on hosted.rs:405 is the only thing keeping this out of the merge queue, because the ruleset requires every thread to be resolved.

The owner question from the Nov 7 comment is still unanswered. When gem or wheel evidence is missing or can't be fetched, should the scan (1) fetch that purl's patch-view record, or (2) treat the purl as unjudgeable (known_stale / drop it from assume_applied)? agent:needs-human was removed today without an answer on the PR, so I haven't picked one. Reply with 1 or 2 and the next agent run will implement it, resolve the thread and enqueue the PR. It stays approved.


Generated by Claude Code

Keeps main's user-patch warnings in the hosted scan's warning list and
drops the record warnings this PR removes. Main's two new cargo VEX gate
tests pass hosted_uuids: Vec::new().

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Owner decision on the hosted.rs:405 review thread. The gem and Python
stale-install probes no longer turn "can't judge" into "not stale":

- Gem: when there is no readable cache/<leaf>.gem (pruned cache, gem
  cleanup) or no patched sha to compare it with, and no agent-mode
  record, the probe fetches that patch's /patches/view record and judges
  the installed dir by its afterHash map.
- Python: when the patched wheel cannot judge a copy (sdist pin, failed
  download or sha256 check, installed copy of another build), the probe
  falls back to the view record the same way, unless another variant
  already proved the copy patched.
- A copy the view cannot judge either (fetch failure, or --offline) is
  reported as unjudged: it leaves assume_applied and goes to a new
  VexBuildParams::unjudged list, which in-run --vex withholds as
  stale_install_unverified, even under --vex-no-verify.

The fallback fetches at most once per uuid and only for copies nothing
else settled, so the normal path still makes no view request. vlt is
unchanged: an undeterminable heal target already lands in stale_purls.
The in-memory engine runs no stale probe, so the Node binding's
fetchPatch stays ignored.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
/code-review high finding: a stale gem dir put only the variant named in
its warning in stale_purls. The other platform variants stayed in
assume_applied, and in-run VEX matches exemptions qualifier-insensitively,
so they re-exempted the stale dir. Also drops a leftover blank line in
the Node smoke test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Owner decision implemented (head 6259cb3). When the gem archive or Python wheel evidence is missing, can't be fetched, or can't judge the installed copy, the stale-install probe fetches that purl's /patches/view record and judges the copy from it. If that fetch also fails, or under --offline, the copy is withheld: it is dropped from assume_applied, and in-run --vex omits it as stale_install_unverified, even with --vex-no-verify. The normal path still makes no view request.

  • Merged main (Read and plan pnpm locks through one core (lock-model 2) #2009, Cargo VEX: read member manifests, match liveness by origin, withhold mixed and vendored bypasses #2012, Never patch a pnpm, yarn or vlt dependency the user patched #2018). The one conflict was the warning list in hosted.rs: I kept main's user-patch warnings and dropped the record warnings this PR removes.
  • vlt was already fail-closed (an undeterminable target goes to stale_purls), so it is unchanged. The in-memory engine runs no stale probe, so fetchPatch stays ignored.
  • /code-review high:
    • Fixed: a stale gem dir now marks every platform variant stale. Before, the qualifier-insensitive VEX exemption could re-exempt it.
    • Declined: a copy judged with no file-level evidence stays "not stale" (main's record semantics: missing files are never evidence). Gem stale purls stay out of known_stale (pre-existing, documented design). Sequential and duplicate view fetches only happen on the fallback path, and both sides fail closed. The fetch still happens when a variant is already stale, because another variant can still prove the dir patched. The VEX helper refactor has different semantics.
  • The hosted.rs:405 thread is replied to and resolved. The PR body is trimmed and updated with the decision.

Generated by Claude Code

@mikolalysenko
Mikola Lysenko (mikolalysenko) added this pull request to the merge queue Nov 9, 2026
Merged via the queue into main with commit c751724 Nov 9, 2026
49 checks passed
@mikolalysenko
Mikola Lysenko (mikolalysenko) deleted the refactor/hosted-scan-no-patch-view branch November 9, 2026 23:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants