Skip to content

fix(release): cap proxied plain-http responses, count Firecrawl depth from the start URL, and drop stale route cache on upgrade - #9860

Merged
waleedlatif1 merged 2 commits into
stagingfrom
fix/release-audit-followups
Nov 10, 2026
Merged

waleedlatif1 merged 2 commits into
stagingfrom
fix/release-audit-followups

Conversation

@waleedlatif1

Copy link
Copy Markdown
Collaborator

Summary

  • Cap guarded responses at maxResponseSize for every dispatcher, not only decoded bodies. undici's ProxyAgent forwards plain-http requests through a client that drops the option, so with HTTP_PROXY set an http:// destination could stream an unbounded body (MCP OAuth fetches, configured endpoints).
  • Count Firecrawl's legacy maxDepth from the starting URL, as v1 did (index files add no depth). Counting from the domain root made the start URL fail its own path filter, so a crawl starting on a deep page returned 0 pages.
  • Clear Next's route cache at container start when an earlier build wrote it. Its keys carry no build id and the image declares it a volume, so a Compose upgrade kept serving ISR pages that point at the old build's chunks.
  • CONTRIBUTING points at npx sim-setup instead of the retired simstudio CLI.

Type of Change

  • Bug fix

Testing

  • New tests fail before each fix: forwarded plain-http response over the cap (environment proxy fixture), deep start URLs with and without path filters, route-cache clearing across builds. Mutation of the build-id check goes red.
  • Bundled bootstrap.ts with the Dockerfile's bun build command and ran it against a fake standalone dir: old page cleared, build marker written, server started.
  • Targeted vitest: firecrawl, route-cache, core/network, input-validation, mistral (206 tests); biome; check:comment-hygiene, check:file-names, check:boundaries, check:utils; regenerated tool metadata and integration docs.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing (new tests pass the test-audit authoring gate)
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

… from the start URL, and drop stale route cache on upgrade
@vercel

vercel Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs Ignored Ignored Preview Nov 10, 2026 11:28pm UTC

Request Review

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 10 files

Reply with feedback, questions, or to request a fix.

Turn on auto-fix | Re-trigger cubic

Comment thread apps/sim/tools/firecrawl/crawl.ts Resolved
@greptile-apps

greptile-apps Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no new actionable issues remain.

Summary

The PR caps plain-HTTP responses sent through a proxy, counts Firecrawl depth from the starting URL, and clears route pages saved by another build.

  • Guarded requests stop reading a response when it exceeds the byte cap.
  • Firecrawl counts crawl depth from the starting page.
  • Container startup clears route pages from an older build.
  • The contributor guide points local setups to the current wizard.

Reviews (2) · Last reviewed commit: "fix(firecrawl): read a scheme-less start..." · Reviewed by Greptile

Comment thread .github/CONTRIBUTING.md Resolved
Comment thread .github/CONTRIBUTING.md Resolved
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 10 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Turn on auto-fix | Re-trigger cubic

@waleedlatif1
waleedlatif1 merged commit a12832d into staging Nov 10, 2026
84 checks passed
@waleedlatif1
waleedlatif1 deleted the fix/release-audit-followups branch November 10, 2026 23:47
waleedlatif1 added a commit that referenced this pull request Nov 10, 2026
Drops the response cap and crawl depth (landed in #9860), CONTRIBUTING (#9860), Firecrawl search sources (on hold), Databricks JSON caps (#9859), and the workflow middleware and route-policies cleanups (#9858). Drops the NO_PROXY entry-syntax docs line (owned by a separate NO_PROXY change). Generated files regenerated on current staging.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant