← WingFoil Benchmark — campaign 13fda3099afa, execution 1
Solo developer: C, D, F
Harness, not model: every arm runs the same agent with the same model, gpt-5.6-terra. Only the harness around it differs: baseline, baseline-docs, wingfoil.
Against the baseline, baseline-docs is better in 1, worse in 2 and the same in 2 of 5 comparisons across categories C, D and F (preliminary: n < 3 in D and F).
Against the baseline, wingfoil is better in 1, worse in 2 and the same in 2 of 5 comparisons across categories C, D and F (preliminary: n < 3 in D and F).
| Category | baseline | baseline-docs | wingfoil wingfoil v0.3.0 · cc31d7a |
|---|---|---|---|
| C — Development S1@1.0 | M-Q1 100.0% n = 3 range 100.0%–100.0% M-K1 0.6929 EUR n = 3 range 0.6352 EUR–0.7702 EUR M-Q1 hold-out 48/48, 48/48, 48/48 n = 3 | M-Q1 99.6% n = 3 range 99.4%–100.0% −0.4 pp ▼ worse within variance M-K1 0.7369 EUR n = 3 range 0.6621 EUR–0.7858 EUR +0.0440 EUR ▼ worse within variance M-Q1 hold-out 45/48, 45/48, 48/48 n = 3 | M-Q1 99.8% n = 3 range 99.4%–100.0% −0.2 pp ▼ worse within variance M-K1 1.3167 EUR n = 3 range 1.1666 EUR–1.4108 EUR +0.6237 EUR ▼ worse beyond variance M-Q1 hold-out 48/48, 46/48, 48/48 n = 3 |
| D — Maintenance & Quality S2@1.0 | M-Q1 95.8% n = 1 preliminary M-D3 1 n = 1 preliminary M-Q1 hold-out 16/17 n = 1 | M-Q1 95.8% n = 1 preliminary ±0.0 pp = same M-D3 1 n = 1 preliminary ±0 = same M-Q1 hold-out 16/17 n = 1 | M-Q1 95.8% n = 1 preliminary ±0.0 pp = same M-D3 1 n = 1 preliminary ±0 = same M-Q1 hold-out 16/17 n = 1 |
| F — Knowledge & Continuity S3@1.0 | M-F1 60.0% n = 1 preliminary M-Q1 hold-out 16/17 n = 1 | M-F1 80.0% n = 1 preliminary +20.0 pp ▲ better M-Q1 hold-out 16/17 n = 1 | M-F1 100.0% n = 1 preliminary +40.0 pp ▲ better M-Q1 hold-out 16/17 n = 1 |
Each value is the mean of its runs, with its n; a single run is preliminary, and so is a comparison unless both sides have n ≥ 3; a range is shown from n = 3. Deltas are against the baseline. How to read these results; which tools are eligible, and why.