WingFoil Benchmark — campaign 33b54dd89e8b, execution 1
Harness, not model: every arm runs the same agent with the same model, claude-sonnet-5-5. Only the harness around it differs: baseline, baseline-docs, wingfoil.
Against the baseline, baseline-docs is better in 2, worse in 4 and the same in 0 of 6 comparisons across categories C, D, E and F.
Against the baseline, wingfoil is better in 4, worse in 2 and the same in 0 of 6 comparisons across categories C, D, E and F.
- Not every harness: only the step-A arms ran, WingFoil against its docs control and the baseline; Spec Kit and OpenSpec wait for their configured variants.
- Step A: the session agent drives WingFoil 0.3.1's workflow, acting in each step's role and stopping at each review gate for the approver; its docs control carries every role's rules.
- Every scenario ran three times on each model, S10 (gates and roles) included, where a separate approver's scripted verdicts are carried out through WingFoil in the wingfoil arm and given as text to the others.
- Approver policy v5 throughout: read arm against arm and model against model; not compared with the executions on policy v1 or v2.
- In S10's wingfoil run r2 on Haiku 5.5 the approver's first verdict was refused by WingFoil (the agent had left an edit uncommitted), so no later verdict was carried out; the arm's manual and verdict hook were corrected afterwards (bug-041, bug-042). No rerun.
- The scenarios' steps are small (a median of about 60 changed lines), so the wingfoil arm's cost at each review gate weighs more than it would on a task of real size.
- Preliminary. The content checks are lexical (bug-026), and S1's step 5 is bimodal in every arm (bug-025).
A, B and G are not covered in this campaign.
| Category | baseline | baseline-docs | wingfoil wingfoil v0.3.1 · 870bd44 |
|---|---|---|---|
| A — Inception & Specification | not covered in this campaign | ||
| B — Planning & Management | not covered in this campaign | ||
| C — Development S1@1.0 | M-Q1 99.8% n = 3 range 99.4%–100.0% M-K1 0.3294 EUR n = 3 range 0.3222 EUR–0.3347 EUR M-Q1 hold-out 48/48, 45/48, 48/48 n = 3 | M-Q1 99.4% n = 3 range 99.4%–99.4% −0.4 pp ▼ worse within variance M-K1 0.4776 EUR n = 3 range 0.4664 EUR–0.4918 EUR +0.1482 EUR ▼ worse beyond variance M-Q1 hold-out 45/48, 45/48, 45/48 n = 3 | M-Q1 99.4% n = 3 range 99.4%–99.4% −0.4 pp ▼ worse within variance M-K1 0.6468 EUR n = 3 range 0.6347 EUR–0.6605 EUR +0.3174 EUR ▼ worse beyond variance M-Q1 hold-out 45/48, 45/48, 45/48 n = 3 |
| D — Maintenance & Quality S2@1.0 | M-Q1 98.6% n = 3 range 95.8%–100.0% M-D3 0.33 n = 3 range 0–1 M-Q1 hold-out 17/17, 17/17, 16/17 n = 3 | M-Q1 97.2% n = 3 range 91.7%–100.0% −1.4 pp ▼ worse within variance M-D3 0.67 n = 3 range 0–2 +0.33 ▼ worse within variance M-Q1 hold-out 16/17, 17/17, 17/17 n = 3 | M-Q1 100.0% n = 3 range 100.0%–100.0% +1.4 pp ▲ better within variance M-D3 0 n = 3 range 0–0 −0.33 ▲ better within variance M-Q1 hold-out 17/17, 17/17, 17/17 n = 3 |
| E — Governance & Compliance S10@1.0 S8@1.0 | S10@1.0 M-E1 not measured M-Q1 hold-out 8/8, 8/8, 6/8 n = 3 S8@1.0 M-E1 8 n = 3 range 7–10 M-Q1 hold-out 7/7, 7/7, 7/7 n = 3 | S10@1.0 M-E1 not measured M-Q1 hold-out 6/8, 6/8, 6/8 n = 3 S8@1.0 M-E1 0 n = 3 range 0–0 −8 ▲ better beyond variance M-Q1 hold-out 7/7, 7/7, 7/7 n = 3 | S10@1.0 M-E1 not measured M-Q1 hold-out 6/8, 8/8, 8/8 n = 3 S8@1.0 M-E1 0 n = 3 range 0–0 −8 ▲ better beyond variance M-Q1 hold-out 6/7, 7/7, 7/7 n = 3 |
| F — Knowledge & Continuity S3@1.0 | M-F1 66.7% n = 3 range 60.0%–80.0% M-Q1 hold-out 16/17, 16/17, 17/17 n = 3 | M-F1 80.0% n = 3 range 80.0%–80.0% +13.3 pp ▲ better within variance M-Q1 hold-out 16/17, 17/17, 17/17 n = 3 | M-F1 80.0% n = 3 range 80.0%–80.0% +13.3 pp ▲ better within variance M-Q1 hold-out 16/17, 17/17, 16/17 n = 3 |
| G — Collaboration | not covered in this campaign | ||
Each value is the mean of its runs, with its n; a single run is preliminary, and so is a comparison unless both sides have n ≥ 3; a range is shown from n = 3. Deltas are against the baseline. How to read these results; which tools are eligible, and why.
Across models
Views by profile
- Solo developer (C, D, F)
- Code reviewer (D, E)
- Team developer (C, F, G)
- Tech lead (B, E, G)
- Non-technical manager (A, B)
- Architect (A, E, F)