Builder pilot: Opus 5.5 rebuilds five app tasks first built by Sonnet 5.5
- Date
- 16 October 2026
- Task type
- Coding
- Status
- Complete
Models tested
| Model or tool | Provider | Open? | How run |
|---|---|---|---|
| Claude Opus 5.5 (builder; rebuilt five tasks) | Anthropic | Closed | Command line |
| Claude Sonnet 5.5 (builder; original accepted builds) | Anthropic | Closed | Command line |
| GPT-6.1 Sol (blind reviewer) | OpenAI | Closed | Command line |
| Muse Wonder Full 1.0 (blind reviewer) | Meta | Closed | Command line |
| Claude Fable 5.1 (checked every reviewer claim) | Anthropic | Closed | Command line |
Purpose
Find out whether Opus 5.5 using fewer tokens would make it cheaper than Sonnet 5.5 on tasks Sonnet could already do.
Context
Five real tasks from app development. The Sonnet side was the build that had originally been accepted. Opus rebuilt each task from the same starting state and brief. Costs are at published list prices.
Method
Two reviewers (GPT-6.1 Sol and Muse Wonder Full) scored each build solo against a rubric (1 to 5) without knowing which model built it, then compared the pair side by side. Fable 5.1 checked each of the 86 claims the reviewers made.
Results
| Measure | Sonnet 5.5 | Opus 5.5 |
|---|---|---|
| Total build cost at list price | $3.78 | $9.02 |
| Task 1 (calls, cost) | 17, $0.72 | 23, $1.91 |
| Task 2 (calls, cost) | 28, $0.685 | 16, $1.60 |
| Task 3 (calls, cost) | 12, $0.36 | 26, $1.32 |
| Task 4 (calls, cost) | 46, $1.07 | 27, $1.60 |
| Task 5 (calls, cost) | 32, $0.94 | 49, $2.59 |
| Mean rubric score, Sol reviewer (1-5) | 3.40 | 3.30 |
| Mean rubric score, Muse reviewer (1-5) | 3.23 | 3.27 |
| Side-by-side preference (10 judgements) | 1 | 9 |
| Verified unique defects | 2 major, 22 minor | 2 major, 18 minor |
Caveats
- One build per task per model (n=1 per cell); the source report calls the result tentative.
- The Opus builds were not taken through review to acceptance; the Sonnet builds were accepted builds.
- The side-by-side judgements favoured the build shown first in 7 of 10 cases, so the 9 of 10 preference for Opus is weak evidence.
- {'Reviewer precision differed': '18 of 24 Sol claims (75%) and 36 of 62 Muse claims (58%) were verified as real.'}
- Checking the reviewers' claims with Fable 5.1 cost $16.38 at list price, more than both builds together. That figure is not in the build costs above.
- Review packets that contained only diffs produced false defect claims.
- What counts as a call in the per-task rows is not defined in the source notes.
What this suggests
On these five tasks Opus cost about 2.4 times as much as Sonnet at list prices and did not score clearly higher on the rubric. The pilot is too small to settle the question of token efficiency.
Source
entries/2026-10-16-builder-pilot-opus-sonnet.md in the GitHub repository (Markdown with YAML frontmatter).