← WingFoil Benchmark — campaign 13fda3099afa, execution 1

C — Development

Does the harness improve how well an agent implements a specification?

S1@1.0

M-Q1: hidden tests passed on the final snapshot (higher is better)

• baseline:
  100.0% n = 3 range 100.0%–100.0%
• baseline-docs:
  99.6% n = 3 range 99.4%–100.0% −0.4 pp ▼ worse within variance
• wingfoil wingfoil v0.3.0 · cc31d7a:
  99.8% n = 3 range 99.4%–100.0% −0.2 pp ▼ worse within variance
  against baseline-docs +0.2 pp ▲ better within variance

M-K1: cost of a run (lower is better)

• baseline:
  0.6929 EUR n = 3 range 0.6352 EUR–0.7702 EUR
• baseline-docs:
  0.7369 EUR n = 3 range 0.6621 EUR–0.7858 EUR +0.0440 EUR ▼ worse within variance
• wingfoil wingfoil v0.3.0 · cc31d7a:
  1.3167 EUR n = 3 range 1.1666 EUR–1.4108 EUR +0.6237 EUR ▼ worse beyond variance
  against baseline-docs +0.5797 EUR ▼ worse beyond variance

Each suite on the final snapshot

• baseline: pointer 12/12, 12/12, 12/12; patch 108/108, 108/108, 108/108; merge-patch 15/15, 15/15, 15/15; create-patch 30/30, 30/30, 30/30
• baseline-docs: pointer 12/12, 12/12, 12/12; patch 108/108, 108/108, 108/108; merge-patch 15/15, 15/15, 15/15; create-patch 29/30, 29/30, 30/30
• wingfoil wingfoil v0.3.0 · cc31d7a: pointer 12/12, 12/12, 12/12; patch 108/108, 108/108, 108/108; merge-patch 15/15, 15/15, 15/15; create-patch 30/30, 29/30, 30/30

Hold-out

• baseline:
  hold-out 48/48, 48/48, 48/48 n = 3
• baseline-docs:
  hold-out 45/48, 45/48, 48/48 n = 3
• wingfoil wingfoil v0.3.0 · cc31d7a:
  hold-out 48/48, 46/48, 48/48 n = 3

Runs

• 13fda3099afa/1/runs/S1@1.0/baseline/gpt-5.6-terra/r1 — open with bench run show 13fda3099afa/1/runs/S1@1.0/baseline/gpt-5.6-terra/r1
• 13fda3099afa/1/runs/S1@1.0/baseline/gpt-5.6-terra/r2 — open with bench run show 13fda3099afa/1/runs/S1@1.0/baseline/gpt-5.6-terra/r2
• 13fda3099afa/1/runs/S1@1.0/baseline/gpt-5.6-terra/r3 — open with bench run show 13fda3099afa/1/runs/S1@1.0/baseline/gpt-5.6-terra/r3
• 13fda3099afa/1/runs/S1@1.0/baseline-docs/gpt-5.6-terra/r1 — open with bench run show 13fda3099afa/1/runs/S1@1.0/baseline-docs/gpt-5.6-terra/r1
• 13fda3099afa/1/runs/S1@1.0/baseline-docs/gpt-5.6-terra/r2 — open with bench run show 13fda3099afa/1/runs/S1@1.0/baseline-docs/gpt-5.6-terra/r2
• 13fda3099afa/1/runs/S1@1.0/baseline-docs/gpt-5.6-terra/r3 — open with bench run show 13fda3099afa/1/runs/S1@1.0/baseline-docs/gpt-5.6-terra/r3
• 13fda3099afa/1/runs/S1@1.0/wingfoil/gpt-5.6-terra/r1 — open with bench run show 13fda3099afa/1/runs/S1@1.0/wingfoil/gpt-5.6-terra/r1
• 13fda3099afa/1/runs/S1@1.0/wingfoil/gpt-5.6-terra/r2 — open with bench run show 13fda3099afa/1/runs/S1@1.0/wingfoil/gpt-5.6-terra/r2
• 13fda3099afa/1/runs/S1@1.0/wingfoil/gpt-5.6-terra/r3 — open with bench run show 13fda3099afa/1/runs/S1@1.0/wingfoil/gpt-5.6-terra/r3

Scenarios that also touch this category

• S3@1.0: primary category F
• S8@1.0: primary category E