← WingFoil Benchmark — campaign c82a5e74885b, execution 2

C — Development

Does the harness improve how well an agent implements a specification?

S1@1.0

M-Q1: hidden tests passed on the final snapshot (higher is better)

• baseline:
  99.8% n = 3 range 99.4%–100.0%
• baseline-docs:
  98.0% n = 3 range 95.8%–99.4% −1.8 pp ▼ worse within variance
• wingfoil:
  98.4% n = 3 range 95.8%–100.0% −1.4 pp ▼ worse within variance

M-K1: cost of a run (lower is better)

• baseline:
  2.4676 EUR n = 3 range 1.7236 EUR–3.7894 EUR
• baseline-docs:
  2.2185 EUR n = 3 range 2.0650 EUR–2.3996 EUR −0.2491 EUR ▲ better within variance
• wingfoil:
  3.8178 EUR n = 3 range 3.6173 EUR–4.1190 EUR +1.3502 EUR ▼ worse within variance

Each suite on the final snapshot

• baseline: pointer 12/12, 12/12, 12/12; patch 108/108, 108/108, 108/108; merge-patch 15/15, 15/15, 15/15; create-patch 29/30, 30/30, 30/30
• baseline-docs: pointer 12/12, 12/12, 12/12; patch 108/108, 108/108, 108/108; merge-patch 15/15, 15/15, 15/15; create-patch 29/30, 28/30, 23/30
• wingfoil: pointer 12/12, 12/12, 12/12; patch 108/108, 108/108, 108/108; merge-patch 15/15, 15/15, 15/15; create-patch 29/30, 23/30, 30/30

Hold-out

• baseline:
  hold-out 45/48, 48/48, 48/48 n = 3
• baseline-docs:
  hold-out 45/48, 45/48, 43/48 n = 3
• wingfoil:
  hold-out 45/48, 43/48, 48/48 n = 3

Runs

• c82a5e74885b/2/runs/S1@1.0/baseline/claude-sonnet-5/r1 — open with bench run show c82a5e74885b/2/runs/S1@1.0/baseline/claude-sonnet-5/r1
• c82a5e74885b/2/runs/S1@1.0/baseline/claude-sonnet-5/r2 — open with bench run show c82a5e74885b/2/runs/S1@1.0/baseline/claude-sonnet-5/r2
• c82a5e74885b/2/runs/S1@1.0/baseline/claude-sonnet-5/r3 — open with bench run show c82a5e74885b/2/runs/S1@1.0/baseline/claude-sonnet-5/r3
• c82a5e74885b/2/runs/S1@1.0/baseline-docs/claude-sonnet-5/r1 — open with bench run show c82a5e74885b/2/runs/S1@1.0/baseline-docs/claude-sonnet-5/r1
• c82a5e74885b/2/runs/S1@1.0/baseline-docs/claude-sonnet-5/r2 — open with bench run show c82a5e74885b/2/runs/S1@1.0/baseline-docs/claude-sonnet-5/r2
• c82a5e74885b/2/runs/S1@1.0/baseline-docs/claude-sonnet-5/r3 — open with bench run show c82a5e74885b/2/runs/S1@1.0/baseline-docs/claude-sonnet-5/r3
• c82a5e74885b/2/runs/S1@1.0/wingfoil/claude-sonnet-5/r1 — open with bench run show c82a5e74885b/2/runs/S1@1.0/wingfoil/claude-sonnet-5/r1
• c82a5e74885b/2/runs/S1@1.0/wingfoil/claude-sonnet-5/r2 — open with bench run show c82a5e74885b/2/runs/S1@1.0/wingfoil/claude-sonnet-5/r2
• c82a5e74885b/2/runs/S1@1.0/wingfoil/claude-sonnet-5/r3 — open with bench run show c82a5e74885b/2/runs/S1@1.0/wingfoil/claude-sonnet-5/r3

Scenarios that also touch this category

• S3@1.0: primary category F
• S8@1.0: primary category E

Other models (slices)

Reported apart from the campaign’s model, and never in its comparisons.

• wingfoil, claude-opus-5: M-Q1 165/165 (n = 1)