← WingFoil Benchmark — campaign 33b54dd89e8b, execution 1
C — Development
Does the harness improve how well an agent implements a specification?
S1@1.0
M-Q1: hidden tests passed on the final snapshot (higher is better)
- baseline: 99.8% n = 3 range 99.4%–100.0%
- baseline-docs: 99.4% n = 3 range 99.4%–99.4% −0.4 pp ▼ worse within variance
- wingfoil wingfoil v0.3.1 · 870bd44: 99.4% n = 3 range 99.4%–99.4% −0.4 pp ▼ worse within varianceagainst baseline-docs ±0.0 pp = same within variance
M-K1: cost of a run (lower is better)
- baseline: 0.3294 EUR n = 3 range 0.3222 EUR–0.3347 EUR
- baseline-docs: 0.4776 EUR n = 3 range 0.4664 EUR–0.4918 EUR +0.1482 EUR ▼ worse beyond variance
- wingfoil wingfoil v0.3.1 · 870bd44: 0.6468 EUR n = 3 range 0.6347 EUR–0.6605 EUR +0.3174 EUR ▼ worse beyond varianceagainst baseline-docs +0.1692 EUR ▼ worse beyond variance
Each suite on the final snapshot
- baseline: pointer 12/12, 12/12, 12/12; patch 108/108, 108/108, 108/108; merge-patch 15/15, 15/15, 15/15; create-patch 30/30, 29/30, 30/30
- baseline-docs: pointer 12/12, 12/12, 12/12; patch 108/108, 108/108, 108/108; merge-patch 15/15, 15/15, 15/15; create-patch 29/30, 29/30, 29/30
- wingfoil wingfoil v0.3.1 · 870bd44: pointer 12/12, 12/12, 12/12; patch 108/108, 108/108, 108/108; merge-patch 15/15, 15/15, 15/15; create-patch 29/30, 29/30, 29/30
Hold-out
- baseline: hold-out 48/48, 45/48, 48/48 n = 3
- baseline-docs: hold-out 45/48, 45/48, 45/48 n = 3
- wingfoil wingfoil v0.3.1 · 870bd44: hold-out 45/48, 45/48, 45/48 n = 3
Runs
33b54dd89e8b/1/runs/S1@1.0/baseline/claude-sonnet-5-5/r1— open withbench run show 33b54dd89e8b/1/runs/S1@1.0/baseline/claude-sonnet-5-5/r133b54dd89e8b/1/runs/S1@1.0/baseline/claude-sonnet-5-5/r2— open withbench run show 33b54dd89e8b/1/runs/S1@1.0/baseline/claude-sonnet-5-5/r233b54dd89e8b/1/runs/S1@1.0/baseline/claude-sonnet-5-5/r3— open withbench run show 33b54dd89e8b/1/runs/S1@1.0/baseline/claude-sonnet-5-5/r333b54dd89e8b/1/runs/S1@1.0/baseline-docs/claude-sonnet-5-5/r1— open withbench run show 33b54dd89e8b/1/runs/S1@1.0/baseline-docs/claude-sonnet-5-5/r133b54dd89e8b/1/runs/S1@1.0/baseline-docs/claude-sonnet-5-5/r2— open withbench run show 33b54dd89e8b/1/runs/S1@1.0/baseline-docs/claude-sonnet-5-5/r233b54dd89e8b/1/runs/S1@1.0/baseline-docs/claude-sonnet-5-5/r3— open withbench run show 33b54dd89e8b/1/runs/S1@1.0/baseline-docs/claude-sonnet-5-5/r333b54dd89e8b/1/runs/S1@1.0/wingfoil/claude-sonnet-5-5/r1— open withbench run show 33b54dd89e8b/1/runs/S1@1.0/wingfoil/claude-sonnet-5-5/r133b54dd89e8b/1/runs/S1@1.0/wingfoil/claude-sonnet-5-5/r2— open withbench run show 33b54dd89e8b/1/runs/S1@1.0/wingfoil/claude-sonnet-5-5/r233b54dd89e8b/1/runs/S1@1.0/wingfoil/claude-sonnet-5-5/r3— open withbench run show 33b54dd89e8b/1/runs/S1@1.0/wingfoil/claude-sonnet-5-5/r3
Scenarios that also touch this category
Other models (slices)
Reported apart from the campaign’s model, and never in its comparisons.
- baseline, claude-haiku-5-5: M-Q1 165/165, 165/165, 165/165 (n = 3)
- baseline-docs, claude-haiku-5-5: M-Q1 165/165, 165/165, 165/165 (n = 3)
- wingfoil wingfoil v0.3.1 · 870bd44, claude-haiku-5-5: M-Q1 164/165, 164/165, 165/165 (n = 3)