← WingFoil Benchmark — campaign 33b54dd89e8b, execution 1
D — Maintenance & Quality
Does the harness improve finding and fixing defects without regressions?
S2@1.0
M-Q1: hidden tests passed on the final snapshot, the defect tests among them (higher is better)
- baseline: 98.6% n = 3 range 95.8%–100.0%
- baseline-docs: 97.2% n = 3 range 91.7%–100.0% −1.4 pp ▼ worse within variance
- wingfoil wingfoil v0.3.1 · 870bd44: 100.0% n = 3 range 100.0%–100.0% +1.4 pp ▲ better within varianceagainst baseline-docs +2.8 pp ▲ better within variance
M-D3: regressions of tests that passed on the seed (lower is better)
- baseline: 0.33 n = 3 range 0–1
- baseline-docs: 0.67 n = 3 range 0–2 +0.33 ▼ worse within variance
- wingfoil wingfoil v0.3.1 · 870bd44: 0 n = 3 range 0–0 −0.33 ▲ better within varianceagainst baseline-docs −0.67 ▲ better within variance
Each suite on the final snapshot
- baseline: regression 16/16, 16/16, 16/16; reports-1 3/3, 3/3, 3/3; reports-2 2/2, 2/2, 2/2; false-report 1/1, 1/1, 0/1; reports-3 2/2, 2/2, 2/2
- baseline-docs: regression 15/16, 16/16, 16/16; reports-1 3/3, 3/3, 3/3; reports-2 2/2, 2/2, 2/2; false-report 0/1, 1/1, 1/1; reports-3 2/2, 2/2, 2/2
- wingfoil wingfoil v0.3.1 · 870bd44: regression 16/16, 16/16, 16/16; reports-1 3/3, 3/3, 3/3; reports-2 2/2, 2/2, 2/2; false-report 1/1, 1/1, 1/1; reports-3 2/2, 2/2, 2/2
Hold-out
- baseline: hold-out 17/17, 17/17, 16/17 n = 3
- baseline-docs: hold-out 16/17, 17/17, 17/17 n = 3
- wingfoil wingfoil v0.3.1 · 870bd44: hold-out 17/17, 17/17, 17/17 n = 3
Runs
33b54dd89e8b/1/runs/S2@1.0/baseline/claude-sonnet-5-5/r1— open withbench run show 33b54dd89e8b/1/runs/S2@1.0/baseline/claude-sonnet-5-5/r133b54dd89e8b/1/runs/S2@1.0/baseline/claude-sonnet-5-5/r2— open withbench run show 33b54dd89e8b/1/runs/S2@1.0/baseline/claude-sonnet-5-5/r233b54dd89e8b/1/runs/S2@1.0/baseline/claude-sonnet-5-5/r3— open withbench run show 33b54dd89e8b/1/runs/S2@1.0/baseline/claude-sonnet-5-5/r333b54dd89e8b/1/runs/S2@1.0/baseline-docs/claude-sonnet-5-5/r1— open withbench run show 33b54dd89e8b/1/runs/S2@1.0/baseline-docs/claude-sonnet-5-5/r133b54dd89e8b/1/runs/S2@1.0/baseline-docs/claude-sonnet-5-5/r2— open withbench run show 33b54dd89e8b/1/runs/S2@1.0/baseline-docs/claude-sonnet-5-5/r233b54dd89e8b/1/runs/S2@1.0/baseline-docs/claude-sonnet-5-5/r3— open withbench run show 33b54dd89e8b/1/runs/S2@1.0/baseline-docs/claude-sonnet-5-5/r333b54dd89e8b/1/runs/S2@1.0/wingfoil/claude-sonnet-5-5/r1— open withbench run show 33b54dd89e8b/1/runs/S2@1.0/wingfoil/claude-sonnet-5-5/r133b54dd89e8b/1/runs/S2@1.0/wingfoil/claude-sonnet-5-5/r2— open withbench run show 33b54dd89e8b/1/runs/S2@1.0/wingfoil/claude-sonnet-5-5/r233b54dd89e8b/1/runs/S2@1.0/wingfoil/claude-sonnet-5-5/r3— open withbench run show 33b54dd89e8b/1/runs/S2@1.0/wingfoil/claude-sonnet-5-5/r3
Scenarios that also touch this category
- S1@1.0: primary category C
Other models (slices)
Reported apart from the campaign’s model, and never in its comparisons.
- baseline, claude-haiku-5-5: M-Q1 24/24, 24/24, 24/24 (n = 3)
- baseline-docs, claude-haiku-5-5: M-Q1 24/24, 24/24, 24/24 (n = 3)
- wingfoil wingfoil v0.3.1 · 870bd44, claude-haiku-5-5: M-Q1 24/24, 24/24, 24/24 (n = 3)