WingFoil Benchmark — campaign 33b54dd89e8b, execution 1

Harness, not model: every arm runs the same agent with the same model, claude-sonnet-5-5. Only the harness around it differs: baseline, baseline-docs, wingfoil.

Against the baseline, baseline-docs is better in 2, worse in 4 and the same in 0 of 6 comparisons across categories C, D, E and F.

Against the baseline, wingfoil is better in 4, worse in 2 and the same in 0 of 6 comparisons across categories C, D, E and F.

A, B and G are not covered in this campaign.

baseline baseline-docs wingfoil (wingfoil v0.3.1 · 870bd44) 100% 50% 0% baseline: 99.8%, n = 3 99.8% n = 3 baseline-docs: 99.4%, n = 3 99.4% n = 3 wingfoil (wingfoil v0.3.1 · 870bd44): 99.4%, n = 3 99.4% n = 3 C · S1@1.0 baseline: 98.6%, n = 3 98.6% n = 3 baseline-docs: 97.2%, n = 3 97.2% n = 3 wingfoil (wingfoil v0.3.1 · 870bd44): 100.0%, n = 3 100.0% n = 3 D · S2@1.0 baseline: 88.1%, n = 3 88.1% n = 3 baseline-docs: 78.6%, n = 3 78.6% n = 3 wingfoil (wingfoil v0.3.1 · 870bd44): 92.9%, n = 3 92.9% n = 3 E · S10@1.0 baseline: 100.0%, n = 3 100.0% n = 3 baseline-docs: 100.0%, n = 3 100.0% n = 3 wingfoil (wingfoil v0.3.1 · 870bd44): 100.0%, n = 3 100.0% n = 3 E · S8@1.0 baseline: 96.4%, n = 3 96.4% n = 3 baseline-docs: 100.0%, n = 3 100.0% n = 3 wingfoil (wingfoil v0.3.1 · 870bd44): 100.0%, n = 3 100.0% n = 3 F · S3@1.0
M-Q1: the share of public hidden tests passed on the final snapshot, per arm, with the range of its runs and their n.
Categorybaselinebaseline-docswingfoil wingfoil v0.3.1 · 870bd44
A — Inception & Specificationnot covered in this campaign
B — Planning & Managementnot covered in this campaign
C — Development
S1@1.0
M-Q1 99.8% n = 3 range 99.4%–100.0%
M-K1 0.3294 EUR n = 3 range 0.3222 EUR–0.3347 EUR
M-Q1 hold-out 48/48, 45/48, 48/48 n = 3
M-Q1 99.4% n = 3 range 99.4%–99.4% −0.4 pp ▼ worse within variance
M-K1 0.4776 EUR n = 3 range 0.4664 EUR–0.4918 EUR +0.1482 EUR ▼ worse beyond variance
M-Q1 hold-out 45/48, 45/48, 45/48 n = 3
M-Q1 99.4% n = 3 range 99.4%–99.4% −0.4 pp ▼ worse within variance
M-K1 0.6468 EUR n = 3 range 0.6347 EUR–0.6605 EUR +0.3174 EUR ▼ worse beyond variance
M-Q1 hold-out 45/48, 45/48, 45/48 n = 3
D — Maintenance & Quality
S2@1.0
M-Q1 98.6% n = 3 range 95.8%–100.0%
M-D3 0.33 n = 3 range 0–1
M-Q1 hold-out 17/17, 17/17, 16/17 n = 3
M-Q1 97.2% n = 3 range 91.7%–100.0% −1.4 pp ▼ worse within variance
M-D3 0.67 n = 3 range 0–2 +0.33 ▼ worse within variance
M-Q1 hold-out 16/17, 17/17, 17/17 n = 3
M-Q1 100.0% n = 3 range 100.0%–100.0% +1.4 pp ▲ better within variance
M-D3 0 n = 3 range 0–0 −0.33 ▲ better within variance
M-Q1 hold-out 17/17, 17/17, 17/17 n = 3
E — Governance & Compliance
S10@1.0
S8@1.0
S10@1.0
M-E1 not measured
M-Q1 hold-out 8/8, 8/8, 6/8 n = 3
S8@1.0
M-E1 8 n = 3 range 7–10
M-Q1 hold-out 7/7, 7/7, 7/7 n = 3
S10@1.0
M-E1 not measured
M-Q1 hold-out 6/8, 6/8, 6/8 n = 3
S8@1.0
M-E1 0 n = 3 range 0–0 −8 ▲ better beyond variance
M-Q1 hold-out 7/7, 7/7, 7/7 n = 3
S10@1.0
M-E1 not measured
M-Q1 hold-out 6/8, 8/8, 8/8 n = 3
S8@1.0
M-E1 0 n = 3 range 0–0 −8 ▲ better beyond variance
M-Q1 hold-out 6/7, 7/7, 7/7 n = 3
F — Knowledge & Continuity
S3@1.0
M-F1 66.7% n = 3 range 60.0%–80.0%
M-Q1 hold-out 16/17, 16/17, 17/17 n = 3
M-F1 80.0% n = 3 range 80.0%–80.0% +13.3 pp ▲ better within variance
M-Q1 hold-out 16/17, 17/17, 17/17 n = 3
M-F1 80.0% n = 3 range 80.0%–80.0% +13.3 pp ▲ better within variance
M-Q1 hold-out 16/17, 17/17, 16/17 n = 3
G — Collaborationnot covered in this campaign

Each value is the mean of its runs, with its n; a single run is preliminary, and so is a comparison unless both sides have n ≥ 3; a range is shown from n = 3. Deltas are against the baseline. How to read these results; which tools are eligible, and why.

Campaign 33b54dd89e8b, execution 1: 45 runs of claude-sonnet-5-5; 45 runs of other models, reported apart.

Across models

Views by profile