← WingFoil Benchmark — campaign 33b54dd89e8b, execution 1
Code reviewer: D, E
Harness, not model: every arm runs the same agent with the same model, claude-sonnet-5-5. Only the harness around it differs: baseline, baseline-docs, wingfoil.
Against the baseline, baseline-docs is better in 1, worse in 2 and the same in 0 of 3 comparisons across categories D and E.
Against the baseline, wingfoil is better in 3, worse in 0 and the same in 0 of 3 comparisons across categories D and E.
| Category | baseline | baseline-docs | wingfoil wingfoil v0.3.1 · 870bd44 |
|---|---|---|---|
| D — Maintenance & Quality S2@1.0 | M-Q1 98.6% n = 3 range 95.8%–100.0% M-D3 0.33 n = 3 range 0–1 M-Q1 hold-out 17/17, 17/17, 16/17 n = 3 | M-Q1 97.2% n = 3 range 91.7%–100.0% −1.4 pp ▼ worse within variance M-D3 0.67 n = 3 range 0–2 +0.33 ▼ worse within variance M-Q1 hold-out 16/17, 17/17, 17/17 n = 3 | M-Q1 100.0% n = 3 range 100.0%–100.0% +1.4 pp ▲ better within variance M-D3 0 n = 3 range 0–0 −0.33 ▲ better within variance M-Q1 hold-out 17/17, 17/17, 17/17 n = 3 |
| E — Governance & Compliance S10@1.0 S8@1.0 | S10@1.0 M-E1 not measured M-Q1 hold-out 8/8, 8/8, 6/8 n = 3 S8@1.0 M-E1 8 n = 3 range 7–10 M-Q1 hold-out 7/7, 7/7, 7/7 n = 3 | S10@1.0 M-E1 not measured M-Q1 hold-out 6/8, 6/8, 6/8 n = 3 S8@1.0 M-E1 0 n = 3 range 0–0 −8 ▲ better beyond variance M-Q1 hold-out 7/7, 7/7, 7/7 n = 3 | S10@1.0 M-E1 not measured M-Q1 hold-out 6/8, 8/8, 8/8 n = 3 S8@1.0 M-E1 0 n = 3 range 0–0 −8 ▲ better beyond variance M-Q1 hold-out 6/7, 7/7, 7/7 n = 3 |
Each value is the mean of its runs, with its n; a single run is preliminary, and so is a comparison unless both sides have n ≥ 3; a range is shown from n = 3. Deltas are against the baseline. How to read these results; which tools are eligible, and why.