← WingFoil Benchmark — campaign 81ebb2691f9c, execution 1

C — Development

Does the harness improve how well an agent implements a specification?

S1@1.0

M-Q1: hidden tests passed on the final snapshot (higher is better)

• baseline:
  99.8% n = 3 range 99.4%–100.0%
• baseline-docs:
  98.2% n = 3 range 95.8%–99.4% −1.6 pp ▼ worse within variance
• openspec openspec 1.14.0 · 9cf16ae:
  99.6% n = 3 range 99.4%–100.0% −0.2 pp ▼ worse within variance
  against openspec-docs +0.2 pp ▲ better within variance
• openspec-docs:
  99.4% n = 3 range 99.4%–99.4% −0.4 pp ▼ worse within variance
• speckit speckit v1.1.0 · f1d3a4f:
  99.6% n = 3 range 99.4%–100.0% −0.2 pp ▼ worse within variance
  against speckit-docs −0.2 pp ▼ worse within variance
• speckit-docs:
  99.8% n = 3 range 99.4%–100.0% ±0.0 pp = same within variance
• wingfoil wingfoil v0.3.0 · cc31d7a:
  99.4% n = 3 range 99.4%–99.4% −0.4 pp ▼ worse within variance
  against baseline-docs +1.2 pp ▲ better within variance

M-K1: cost of a run (lower is better)

• baseline:
  0.3454 EUR n = 3 range 0.3209 EUR–0.3703 EUR
• baseline-docs:
  0.4609 EUR n = 3 range 0.4388 EUR–0.4819 EUR +0.1154 EUR ▼ worse beyond variance
• openspec openspec 1.14.0 · 9cf16ae:
  0.8458 EUR n = 3 range 0.7710 EUR–0.9043 EUR +0.5004 EUR ▼ worse beyond variance
  against openspec-docs +0.4824 EUR ▼ worse beyond variance
• openspec-docs:
  0.3634 EUR n = 3 range 0.3567 EUR–0.3716 EUR +0.0180 EUR ▼ worse within variance
• speckit speckit v1.1.0 · f1d3a4f:
  0.5114 EUR n = 3 range 0.4840 EUR–0.5419 EUR +0.1660 EUR ▼ worse beyond variance
  against speckit-docs +0.1613 EUR ▼ worse beyond variance
• speckit-docs:
  0.3501 EUR n = 3 range 0.3201 EUR–0.3855 EUR +0.0046 EUR ▼ worse within variance
• wingfoil wingfoil v0.3.0 · cc31d7a:
  0.5857 EUR n = 3 range 0.5556 EUR–0.6013 EUR +0.2402 EUR ▼ worse beyond variance
  against baseline-docs +0.1248 EUR ▼ worse beyond variance

Each suite on the final snapshot

• baseline: pointer 12/12, 12/12, 12/12; patch 108/108, 108/108, 108/108; merge-patch 15/15, 15/15, 15/15; create-patch 29/30, 30/30, 30/30
• baseline-docs: pointer 12/12, 12/12, 12/12; patch 108/108, 108/108, 108/108; merge-patch 15/15, 15/15, 15/15; create-patch 29/30, 23/30, 29/30
• openspec openspec 1.14.0 · 9cf16ae: pointer 12/12, 12/12, 12/12; patch 108/108, 108/108, 108/108; merge-patch 15/15, 15/15, 15/15; create-patch 30/30, 29/30, 29/30
• openspec-docs: pointer 12/12, 12/12, 12/12; patch 108/108, 108/108, 108/108; merge-patch 15/15, 15/15, 15/15; create-patch 29/30, 29/30, 29/30
• speckit speckit v1.1.0 · f1d3a4f: pointer 12/12, 12/12, 12/12; patch 108/108, 108/108, 108/108; merge-patch 15/15, 15/15, 15/15; create-patch 29/30, 29/30, 30/30
• speckit-docs: pointer 12/12, 12/12, 12/12; patch 108/108, 108/108, 108/108; merge-patch 15/15, 15/15, 15/15; create-patch 29/30, 30/30, 30/30
• wingfoil wingfoil v0.3.0 · cc31d7a: pointer 12/12, 12/12, 12/12; patch 108/108, 108/108, 108/108; merge-patch 15/15, 15/15, 15/15; create-patch 29/30, 29/30, 29/30

Hold-out

• baseline:
  hold-out 45/48, 48/48, 48/48 n = 3
• baseline-docs:
  hold-out 45/48, 43/48, 45/48 n = 3
• openspec openspec 1.14.0 · 9cf16ae:
  hold-out 48/48, 45/48, 45/48 n = 3
• openspec-docs:
  hold-out 45/48, 45/48, 45/48 n = 3
• speckit speckit v1.1.0 · f1d3a4f:
  hold-out 45/48, 45/48, 48/48 n = 3
• speckit-docs:
  hold-out 45/48, 48/48, 48/48 n = 3
• wingfoil wingfoil v0.3.0 · cc31d7a:
  hold-out 45/48, 45/48, 45/48 n = 3

Runs

• 81ebb2691f9c/1/runs/S1@1.0/baseline/claude-sonnet-5-5/r1 — open with bench run show 81ebb2691f9c/1/runs/S1@1.0/baseline/claude-sonnet-5-5/r1
• 81ebb2691f9c/1/runs/S1@1.0/baseline/claude-sonnet-5-5/r2 — open with bench run show 81ebb2691f9c/1/runs/S1@1.0/baseline/claude-sonnet-5-5/r2
• 81ebb2691f9c/1/runs/S1@1.0/baseline/claude-sonnet-5-5/r3 — open with bench run show 81ebb2691f9c/1/runs/S1@1.0/baseline/claude-sonnet-5-5/r3
• 81ebb2691f9c/1/runs/S1@1.0/baseline-docs/claude-sonnet-5-5/r1 — open with bench run show 81ebb2691f9c/1/runs/S1@1.0/baseline-docs/claude-sonnet-5-5/r1
• 81ebb2691f9c/1/runs/S1@1.0/baseline-docs/claude-sonnet-5-5/r2 — open with bench run show 81ebb2691f9c/1/runs/S1@1.0/baseline-docs/claude-sonnet-5-5/r2
• 81ebb2691f9c/1/runs/S1@1.0/baseline-docs/claude-sonnet-5-5/r3 — open with bench run show 81ebb2691f9c/1/runs/S1@1.0/baseline-docs/claude-sonnet-5-5/r3
• 81ebb2691f9c/1/runs/S1@1.0/openspec/claude-sonnet-5-5/r1 — open with bench run show 81ebb2691f9c/1/runs/S1@1.0/openspec/claude-sonnet-5-5/r1
• 81ebb2691f9c/1/runs/S1@1.0/openspec/claude-sonnet-5-5/r2 — open with bench run show 81ebb2691f9c/1/runs/S1@1.0/openspec/claude-sonnet-5-5/r2
• 81ebb2691f9c/1/runs/S1@1.0/openspec/claude-sonnet-5-5/r3 — open with bench run show 81ebb2691f9c/1/runs/S1@1.0/openspec/claude-sonnet-5-5/r3
• 81ebb2691f9c/1/runs/S1@1.0/openspec-docs/claude-sonnet-5-5/r1 — open with bench run show 81ebb2691f9c/1/runs/S1@1.0/openspec-docs/claude-sonnet-5-5/r1
• 81ebb2691f9c/1/runs/S1@1.0/openspec-docs/claude-sonnet-5-5/r2 — open with bench run show 81ebb2691f9c/1/runs/S1@1.0/openspec-docs/claude-sonnet-5-5/r2
• 81ebb2691f9c/1/runs/S1@1.0/openspec-docs/claude-sonnet-5-5/r3 — open with bench run show 81ebb2691f9c/1/runs/S1@1.0/openspec-docs/claude-sonnet-5-5/r3
• 81ebb2691f9c/1/runs/S1@1.0/speckit/claude-sonnet-5-5/r1 — open with bench run show 81ebb2691f9c/1/runs/S1@1.0/speckit/claude-sonnet-5-5/r1
• 81ebb2691f9c/1/runs/S1@1.0/speckit/claude-sonnet-5-5/r2 — open with bench run show 81ebb2691f9c/1/runs/S1@1.0/speckit/claude-sonnet-5-5/r2
• 81ebb2691f9c/1/runs/S1@1.0/speckit/claude-sonnet-5-5/r3 — open with bench run show 81ebb2691f9c/1/runs/S1@1.0/speckit/claude-sonnet-5-5/r3
• 81ebb2691f9c/1/runs/S1@1.0/speckit-docs/claude-sonnet-5-5/r1 — open with bench run show 81ebb2691f9c/1/runs/S1@1.0/speckit-docs/claude-sonnet-5-5/r1
• 81ebb2691f9c/1/runs/S1@1.0/speckit-docs/claude-sonnet-5-5/r2 — open with bench run show 81ebb2691f9c/1/runs/S1@1.0/speckit-docs/claude-sonnet-5-5/r2
• 81ebb2691f9c/1/runs/S1@1.0/speckit-docs/claude-sonnet-5-5/r3 — open with bench run show 81ebb2691f9c/1/runs/S1@1.0/speckit-docs/claude-sonnet-5-5/r3
• 81ebb2691f9c/1/runs/S1@1.0/wingfoil/claude-sonnet-5-5/r1 — open with bench run show 81ebb2691f9c/1/runs/S1@1.0/wingfoil/claude-sonnet-5-5/r1
• 81ebb2691f9c/1/runs/S1@1.0/wingfoil/claude-sonnet-5-5/r2 — open with bench run show 81ebb2691f9c/1/runs/S1@1.0/wingfoil/claude-sonnet-5-5/r2
• 81ebb2691f9c/1/runs/S1@1.0/wingfoil/claude-sonnet-5-5/r3 — open with bench run show 81ebb2691f9c/1/runs/S1@1.0/wingfoil/claude-sonnet-5-5/r3

Scenarios that also touch this category

• S3@1.0: primary category F
• S8@1.0: primary category E

Other models (slices)

Reported apart from the campaign’s model, and never in its comparisons.

• baseline, claude-haiku-5-5: M-Q1 164/165 (n = 1)
• baseline, claude-opus-5-5: M-Q1 165/165 (n = 1)
• baseline-docs, claude-opus-5-5: M-Q1 165/165 (n = 1)
• wingfoil wingfoil v0.3.0 · cc31d7a, claude-haiku-5-5: M-Q1 165/165 (n = 1)
• wingfoil wingfoil v0.3.0 · cc31d7a, claude-opus-5-5: M-Q1 165/165 (n = 1)