← WingFoil Benchmark — campaign 697e79b75e41, execution 1
D — Maintenance & Quality
Does the harness improve finding and fixing defects without regressions?
S2@1.0
M-Q1: hidden tests passed on the final snapshot, the defect tests among them (higher is better)
- baseline: 95.8% n = 1 preliminary
- baseline-docs: 95.8% n = 1 preliminary ±0.0 pp = same
- wingfoil wingfoil v0.3.1 · 870bd44: 95.8% n = 1 preliminary ±0.0 pp = sameagainst baseline-docs ±0.0 pp = same
M-D3: regressions of tests that passed on the seed (lower is better)
- baseline: 1 n = 1 preliminary
- baseline-docs: 1 n = 1 preliminary ±0 = same
- wingfoil wingfoil v0.3.1 · 870bd44: 1 n = 1 preliminary ±0 = sameagainst baseline-docs ±0 = same
Each suite on the final snapshot
- baseline: regression 16/16; reports-1 3/3; reports-2 2/2; false-report 0/1; reports-3 2/2
- baseline-docs: regression 16/16; reports-1 3/3; reports-2 2/2; false-report 0/1; reports-3 2/2
- wingfoil wingfoil v0.3.1 · 870bd44: regression 16/16; reports-1 3/3; reports-2 2/2; false-report 0/1; reports-3 2/2
Hold-out
- baseline: hold-out 16/17 n = 1
- baseline-docs: hold-out 16/17 n = 1
- wingfoil wingfoil v0.3.1 · 870bd44: hold-out 16/17 n = 1
Runs
697e79b75e41/1/runs/S2@1.0/baseline/gpt-5.6-terra/r1— open withbench run show 697e79b75e41/1/runs/S2@1.0/baseline/gpt-5.6-terra/r1697e79b75e41/1/runs/S2@1.0/baseline-docs/gpt-5.6-terra/r1— open withbench run show 697e79b75e41/1/runs/S2@1.0/baseline-docs/gpt-5.6-terra/r1697e79b75e41/1/runs/S2@1.0/wingfoil/gpt-5.6-terra/r1— open withbench run show 697e79b75e41/1/runs/S2@1.0/wingfoil/gpt-5.6-terra/r1
Scenarios that also touch this category
- S1@1.0: primary category C
Other models (slices)
Reported apart from the campaign’s model, and never in its comparisons.
- baseline, gpt-6-luna: M-Q1 23/24, 23/24, 23/24 (n = 3)
- baseline-docs, gpt-6-luna: M-Q1 23/24, 23/24, 23/24 (n = 3)
- wingfoil wingfoil v0.3.1 · 870bd44, gpt-6-luna: M-Q1 23/24, 23/24, 22/24 (n = 3)