← WingFoil Benchmark — campaign 81ebb2691f9c, execution 1
Code reviewer: D, E
Harness, not model: every arm runs the same agent with the same model, claude-sonnet-5-5. Only the harness around it differs: baseline, baseline-docs, openspec, openspec-docs, speckit, speckit-docs, wingfoil.
Against the baseline, baseline-docs is better in 1, worse in 0 and the same in 2 of 3 comparisons across categories D and E (preliminary: n < 3 in D and E).
Against the baseline, openspec is better in 1, worse in 0 and the same in 2 of 3 comparisons across categories D and E (preliminary: n < 3 in D and E).
Against the baseline, openspec-docs is better in 1, worse in 0 and the same in 2 of 3 comparisons across categories D and E (preliminary: n < 3 in D and E).
Against the baseline, speckit is better in 1, worse in 0 and the same in 2 of 3 comparisons across categories D and E (preliminary: n < 3 in D and E).
Against the baseline, speckit-docs is better in 1, worse in 0 and the same in 2 of 3 comparisons across categories D and E (preliminary: n < 3 in D and E).
Against the baseline, wingfoil is better in 1, worse in 0 and the same in 2 of 3 comparisons across categories D and E (preliminary: n < 3 in D and E).
| Category | baseline | baseline-docs | openspec openspec 1.14.0 · 9cf16ae | openspec-docs | speckit speckit v1.1.0 · f1d3a4f | speckit-docs | wingfoil wingfoil v0.3.0 · cc31d7a |
|---|---|---|---|---|---|---|---|
| D — Maintenance & Quality S2@1.0 | M-Q1 100.0% n = 1 preliminary M-D3 0 n = 1 preliminary M-Q1 hold-out 17/17 n = 1 | M-Q1 100.0% n = 1 preliminary ±0.0 pp = same M-D3 0 n = 1 preliminary ±0 = same M-Q1 hold-out 17/17 n = 1 | M-Q1 100.0% n = 1 preliminary ±0.0 pp = same M-D3 0 n = 1 preliminary ±0 = same M-Q1 hold-out 17/17 n = 1 | M-Q1 100.0% n = 1 preliminary ±0.0 pp = same M-D3 0 n = 1 preliminary ±0 = same M-Q1 hold-out 17/17 n = 1 | M-Q1 100.0% n = 1 preliminary ±0.0 pp = same M-D3 0 n = 1 preliminary ±0 = same M-Q1 hold-out 17/17 n = 1 | M-Q1 100.0% n = 1 preliminary ±0.0 pp = same M-D3 0 n = 1 preliminary ±0 = same M-Q1 hold-out 17/17 n = 1 | M-Q1 100.0% n = 1 preliminary ±0.0 pp = same M-D3 0 n = 1 preliminary ±0 = same M-Q1 hold-out 17/17 n = 1 |
| E — Governance & Compliance S8@1.0 | M-E1 7 n = 1 preliminary M-Q1 hold-out 7/7 n = 1 | M-E1 0 n = 1 preliminary −7 ▲ better M-Q1 hold-out 7/7 n = 1 | M-E1 0 n = 1 preliminary −7 ▲ better M-Q1 hold-out 6/7 n = 1 | M-E1 0 n = 1 preliminary −7 ▲ better M-Q1 hold-out 7/7 n = 1 | M-E1 0 n = 1 preliminary −7 ▲ better M-Q1 hold-out 7/7 n = 1 | M-E1 0 n = 1 preliminary −7 ▲ better M-Q1 hold-out 7/7 n = 1 | M-E1 0 n = 1 preliminary −7 ▲ better M-Q1 hold-out 7/7 n = 1 |
Each value is the mean of its runs, with its n; a single run is preliminary, and so is a comparison unless both sides have n ≥ 3; a range is shown from n = 3. Deltas are against the baseline. How to read these results; which tools are eligible, and why.