← WingFoil Benchmark — campaign 697e79b75e41, execution 1
C — Development
Does the harness improve how well an agent implements a specification?
S1@1.0
M-Q1: hidden tests passed on the final snapshot (higher is better)
- baseline: 98.4% n = 3 range 96.4%–99.4%
- baseline-docs: 99.4% n = 3 range 99.4%–99.4% +1.0 pp ▲ better within variance
- wingfoil wingfoil v0.3.1 · 870bd44: 99.8% n = 3 range 99.4%–100.0% +1.4 pp ▲ better within varianceagainst baseline-docs +0.4 pp ▲ better within variance
M-K1: cost of a run (lower is better)
- baseline: 0.7278 EUR n = 3 range 0.6958 EUR–0.7504 EUR
- baseline-docs: 0.8237 EUR n = 3 range 0.7888 EUR–0.8635 EUR +0.0959 EUR ▼ worse beyond variance
- wingfoil wingfoil v0.3.1 · 870bd44: 1.1898 EUR n = 3 range 1.0827 EUR–1.2612 EUR +0.4619 EUR ▼ worse beyond varianceagainst baseline-docs +0.3660 EUR ▼ worse beyond variance
Each suite on the final snapshot
- baseline: pointer 12/12, 12/12, 12/12; patch 108/108, 102/108, 108/108; merge-patch 15/15, 15/15, 15/15; create-patch 29/30, 30/30, 29/30
- baseline-docs: pointer 12/12, 12/12, 12/12; patch 108/108, 108/108, 108/108; merge-patch 15/15, 15/15, 15/15; create-patch 29/30, 29/30, 29/30
- wingfoil wingfoil v0.3.1 · 870bd44: pointer 12/12, 12/12, 12/12; patch 108/108, 108/108, 108/108; merge-patch 15/15, 15/15, 15/15; create-patch 30/30, 29/30, 30/30
Hold-out
- baseline: hold-out 45/48, 48/48, 45/48 n = 3
- baseline-docs: hold-out 45/48, 46/48, 45/48 n = 3
- wingfoil wingfoil v0.3.1 · 870bd44: hold-out 48/48, 46/48, 48/48 n = 3
Runs
697e79b75e41/1/runs/S1@1.0/baseline/gpt-5.6-terra/r1— open withbench run show 697e79b75e41/1/runs/S1@1.0/baseline/gpt-5.6-terra/r1697e79b75e41/1/runs/S1@1.0/baseline/gpt-5.6-terra/r2— open withbench run show 697e79b75e41/1/runs/S1@1.0/baseline/gpt-5.6-terra/r2697e79b75e41/1/runs/S1@1.0/baseline/gpt-5.6-terra/r3— open withbench run show 697e79b75e41/1/runs/S1@1.0/baseline/gpt-5.6-terra/r3697e79b75e41/1/runs/S1@1.0/baseline-docs/gpt-5.6-terra/r1— open withbench run show 697e79b75e41/1/runs/S1@1.0/baseline-docs/gpt-5.6-terra/r1697e79b75e41/1/runs/S1@1.0/baseline-docs/gpt-5.6-terra/r2— open withbench run show 697e79b75e41/1/runs/S1@1.0/baseline-docs/gpt-5.6-terra/r2697e79b75e41/1/runs/S1@1.0/baseline-docs/gpt-5.6-terra/r3— open withbench run show 697e79b75e41/1/runs/S1@1.0/baseline-docs/gpt-5.6-terra/r3— 2 attempts697e79b75e41/1/runs/S1@1.0/wingfoil/gpt-5.6-terra/r1— open withbench run show 697e79b75e41/1/runs/S1@1.0/wingfoil/gpt-5.6-terra/r1697e79b75e41/1/runs/S1@1.0/wingfoil/gpt-5.6-terra/r2— open withbench run show 697e79b75e41/1/runs/S1@1.0/wingfoil/gpt-5.6-terra/r2697e79b75e41/1/runs/S1@1.0/wingfoil/gpt-5.6-terra/r3— open withbench run show 697e79b75e41/1/runs/S1@1.0/wingfoil/gpt-5.6-terra/r3
Scenarios that also touch this category
Other models (slices)
Reported apart from the campaign’s model, and never in its comparisons.
- baseline, gpt-6-luna: M-Q1 149/165, 164/165, 163/165 (n = 3)
- baseline-docs, gpt-6-luna: M-Q1 105/165, 165/165, 158/165 (n = 3)
- wingfoil wingfoil v0.3.1 · 870bd44, gpt-6-luna: M-Q1 164/165, 164/165, 164/165 (n = 3)