Skip to content

eval: desk13 Exact (contract) Opus probe in our harness - #61

Open
ryansolid wants to merge 3 commits into
mainfrom
claude/project-thread-xp2irv
Open

ryansolid wants to merge 3 commits into
mainfrom
claude/project-thread-xp2irv

Conversation

@ryansolid

Copy link
Copy Markdown
Member

Adds the write-up and grades for three Opus apps built in Exact (the contract arm from #59), run the way we ran the .rx probe. See eval/results/desk13-exact-2026-11-13/README.md.

  • Defects: 1, 2 and 0, a mean of 1.0. The last three .rx apps scored 2, 0 and 0 (mean 0.67). With three apps per arm, neither difference is meaningful.
  • Isolation: agents worked from a copy of Exact with bench/ removed, so they could not read Exact's desk9 reference.
  • Shared defect: two of the three apps left the settings save on Exact's default mutation mode instead of parallel. A later save then replaces the earlier one, so the earlier result never shows.
  • Cost: Exact apps used 145–168k tokens and 10–12 minutes each, against 99–107k tokens and about 5 minutes for .rx.
  • Size: 568–605 lines per Exact app, against 411–452 for .rx.

Generated by Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt
…en, agent wall time from .agent.json)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWLtkg281f1NkGEDjCt

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants