← WingFoil Benchmark — campaign 697e79b75e41, execution 1

Solo developer: C, D, F

Harness, not model: every arm runs the same agent with the same model, gpt-5.6-terra. Only the harness around it differs: baseline, baseline-docs, wingfoil.

Against the baseline, baseline-docs is better in 2, worse in 1 and the same in 2 of 5 comparisons across categories C, D and F (preliminary: n < 3 in D and F).

Against the baseline, wingfoil is better in 1, worse in 1 and the same in 3 of 5 comparisons across categories C, D and F (preliminary: n < 3 in D and F).

baseline baseline-docs wingfoil (wingfoil v0.3.1 · 870bd44) 100% 50% 0% baseline: 98.4%, n = 3 98.4% n = 3 baseline-docs: 99.4%, n = 3 99.4% n = 3 wingfoil (wingfoil v0.3.1 · 870bd44): 99.8%, n = 3 99.8% n = 3 C · S1@1.0 baseline: 95.8%, n = 1 95.8% n = 1 baseline-docs: 95.8%, n = 1 95.8% n = 1 wingfoil (wingfoil v0.3.1 · 870bd44): 95.8%, n = 1 95.8% n = 1 D · S2@1.0 baseline: 91.9%, n = 1 91.9% n = 1 baseline-docs: 100.0%, n = 1 100.0% n = 1 wingfoil (wingfoil v0.3.1 · 870bd44): 91.9%, n = 1 91.9% n = 1 F · S3@1.0
M-Q1: the share of public hidden tests passed on the final snapshot, per arm, with the range of its runs and their n.
Categorybaselinebaseline-docswingfoil wingfoil v0.3.1 · 870bd44
C — Development
S1@1.0
M-Q1 98.4% n = 3 range 96.4%–99.4%
M-K1 0.7278 EUR n = 3 range 0.6958 EUR–0.7504 EUR
M-Q1 hold-out 45/48, 48/48, 45/48 n = 3
M-Q1 99.4% n = 3 range 99.4%–99.4% +1.0 pp ▲ better within variance
M-K1 0.8237 EUR n = 3 range 0.7888 EUR–0.8635 EUR +0.0959 EUR ▼ worse beyond variance
M-Q1 hold-out 45/48, 46/48, 45/48 n = 3
M-Q1 99.8% n = 3 range 99.4%–100.0% +1.4 pp ▲ better within variance
M-K1 1.1898 EUR n = 3 range 1.0827 EUR–1.2612 EUR +0.4619 EUR ▼ worse beyond variance
M-Q1 hold-out 48/48, 46/48, 48/48 n = 3
D — Maintenance & Quality
S2@1.0
M-Q1 95.8% n = 1 preliminary
M-D3 1 n = 1 preliminary
M-Q1 hold-out 16/17 n = 1
M-Q1 95.8% n = 1 preliminary ±0.0 pp = same
M-D3 1 n = 1 preliminary ±0 = same
M-Q1 hold-out 16/17 n = 1
M-Q1 95.8% n = 1 preliminary ±0.0 pp = same
M-D3 1 n = 1 preliminary ±0 = same
M-Q1 hold-out 16/17 n = 1
F — Knowledge & Continuity
S3@1.0
M-F1 60.0% n = 1 preliminary
M-Q1 hold-out 16/17 n = 1
M-F1 80.0% n = 1 preliminary +20.0 pp ▲ better
M-Q1 hold-out 16/17 n = 1
M-F1 60.0% n = 1 preliminary ±0.0 pp = same
M-Q1 hold-out 16/17 n = 1

Each value is the mean of its runs, with its n; a single run is preliminary, and so is a comparison unless both sides have n ≥ 3; a range is shown from n = 3. Deltas are against the baseline. How to read these results; which tools are eligible, and why.

Campaign 697e79b75e41, execution 1: 15 runs of gpt-5.6-terra; 27 runs of other models, reported apart.