WingFoil Benchmark — campaign 81ebb2691f9c, execution 1
Harness, not model: every arm runs the same agent with the same model, claude-sonnet-5-5. Only the harness around it differs: baseline, baseline-docs, openspec, openspec-docs, speckit, speckit-docs, wingfoil.
Against the baseline, baseline-docs is better in 2, worse in 2 and the same in 2 of 6 comparisons across categories C, D, E and F (preliminary: n < 3 in D, E and F).
Against the baseline, openspec is better in 2, worse in 2 and the same in 2 of 6 comparisons across categories C, D, E and F (preliminary: n < 3 in D, E and F).
Against the baseline, openspec-docs is better in 1, worse in 2 and the same in 3 of 6 comparisons across categories C, D, E and F (preliminary: n < 3 in D, E and F).
Against the baseline, speckit is better in 1, worse in 2 and the same in 3 of 6 comparisons across categories C, D, E and F (preliminary: n < 3 in D, E and F).
Against the baseline, speckit-docs is better in 1, worse in 1 and the same in 4 of 6 comparisons across categories C, D, E and F (preliminary: n < 3 in D, E and F).
Against the baseline, wingfoil is better in 1, worse in 2 and the same in 3 of 6 comparisons across categories C, D, E and F (preliminary: n < 3 in D, E and F).
A, B and G are not covered in this campaign.
| Category | baseline | baseline-docs | openspec openspec 1.14.0 · 9cf16ae | openspec-docs | speckit speckit v1.1.0 · f1d3a4f | speckit-docs | wingfoil wingfoil v0.3.0 · cc31d7a |
|---|---|---|---|---|---|---|---|
| A — Inception & Specification | not covered in this campaign | ||||||
| B — Planning & Management | not covered in this campaign | ||||||
| C — Development S1@1.0 | M-Q1 99.8% n = 3 range 99.4%–100.0% M-K1 0.3454 EUR n = 3 range 0.3209 EUR–0.3703 EUR M-Q1 hold-out 45/48, 48/48, 48/48 n = 3 | M-Q1 98.2% n = 3 range 95.8%–99.4% −1.6 pp ▼ worse within variance M-K1 0.4609 EUR n = 3 range 0.4388 EUR–0.4819 EUR +0.1154 EUR ▼ worse beyond variance M-Q1 hold-out 45/48, 43/48, 45/48 n = 3 | M-Q1 99.6% n = 3 range 99.4%–100.0% −0.2 pp ▼ worse within variance M-K1 0.8458 EUR n = 3 range 0.7710 EUR–0.9043 EUR +0.5004 EUR ▼ worse beyond variance M-Q1 hold-out 48/48, 45/48, 45/48 n = 3 | M-Q1 99.4% n = 3 range 99.4%–99.4% −0.4 pp ▼ worse within variance M-K1 0.3634 EUR n = 3 range 0.3567 EUR–0.3716 EUR +0.0180 EUR ▼ worse within variance M-Q1 hold-out 45/48, 45/48, 45/48 n = 3 | M-Q1 99.6% n = 3 range 99.4%–100.0% −0.2 pp ▼ worse within variance M-K1 0.5114 EUR n = 3 range 0.4840 EUR–0.5419 EUR +0.1660 EUR ▼ worse beyond variance M-Q1 hold-out 45/48, 45/48, 48/48 n = 3 | M-Q1 99.8% n = 3 range 99.4%–100.0% ±0.0 pp = same within variance M-K1 0.3501 EUR n = 3 range 0.3201 EUR–0.3855 EUR +0.0046 EUR ▼ worse within variance M-Q1 hold-out 45/48, 48/48, 48/48 n = 3 | M-Q1 99.4% n = 3 range 99.4%–99.4% −0.4 pp ▼ worse within variance M-K1 0.5857 EUR n = 3 range 0.5556 EUR–0.6013 EUR +0.2402 EUR ▼ worse beyond variance M-Q1 hold-out 45/48, 45/48, 45/48 n = 3 |
| D — Maintenance & Quality S2@1.0 | M-Q1 100.0% n = 1 preliminary M-D3 0 n = 1 preliminary M-Q1 hold-out 17/17 n = 1 | M-Q1 100.0% n = 1 preliminary ±0.0 pp = same M-D3 0 n = 1 preliminary ±0 = same M-Q1 hold-out 17/17 n = 1 | M-Q1 100.0% n = 1 preliminary ±0.0 pp = same M-D3 0 n = 1 preliminary ±0 = same M-Q1 hold-out 17/17 n = 1 | M-Q1 100.0% n = 1 preliminary ±0.0 pp = same M-D3 0 n = 1 preliminary ±0 = same M-Q1 hold-out 17/17 n = 1 | M-Q1 100.0% n = 1 preliminary ±0.0 pp = same M-D3 0 n = 1 preliminary ±0 = same M-Q1 hold-out 17/17 n = 1 | M-Q1 100.0% n = 1 preliminary ±0.0 pp = same M-D3 0 n = 1 preliminary ±0 = same M-Q1 hold-out 17/17 n = 1 | M-Q1 100.0% n = 1 preliminary ±0.0 pp = same M-D3 0 n = 1 preliminary ±0 = same M-Q1 hold-out 17/17 n = 1 |
| E — Governance & Compliance S8@1.0 | M-E1 7 n = 1 preliminary M-Q1 hold-out 7/7 n = 1 | M-E1 0 n = 1 preliminary −7 ▲ better M-Q1 hold-out 7/7 n = 1 | M-E1 0 n = 1 preliminary −7 ▲ better M-Q1 hold-out 6/7 n = 1 | M-E1 0 n = 1 preliminary −7 ▲ better M-Q1 hold-out 7/7 n = 1 | M-E1 0 n = 1 preliminary −7 ▲ better M-Q1 hold-out 7/7 n = 1 | M-E1 0 n = 1 preliminary −7 ▲ better M-Q1 hold-out 7/7 n = 1 | M-E1 0 n = 1 preliminary −7 ▲ better M-Q1 hold-out 7/7 n = 1 |
| F — Knowledge & Continuity S3@1.0 | M-F1 60.0% n = 1 preliminary M-Q1 hold-out 16/17 n = 1 | M-F1 80.0% n = 1 preliminary +20.0 pp ▲ better M-Q1 hold-out 17/17 n = 1 | M-F1 100.0% n = 1 preliminary +40.0 pp ▲ better M-Q1 hold-out 17/17 n = 1 | M-F1 60.0% n = 1 preliminary ±0.0 pp = same M-Q1 hold-out 16/17 n = 1 | M-F1 60.0% n = 1 preliminary ±0.0 pp = same M-Q1 hold-out 15/17 n = 1 | M-F1 60.0% n = 1 preliminary ±0.0 pp = same M-Q1 hold-out 16/17 n = 1 | M-F1 60.0% n = 1 preliminary ±0.0 pp = same M-Q1 hold-out 16/17 n = 1 |
| G — Collaboration | not covered in this campaign | ||||||
Each value is the mean of its runs, with its n; a single run is preliminary, and so is a comparison unless both sides have n ≥ 3; a range is shown from n = 3. Deltas are against the baseline. How to read these results; which tools are eligible, and why.
Compared with earlier executions
Views by profile
- Solo developer (C, D, F)
- Code reviewer (D, E)
- Team developer (C, F, G)
- Tech lead (B, E, G)
- Non-technical manager (A, B)
- Architect (A, E, F)