WingFoil Benchmark — campaign 697e79b75e41, execution 1
Harness, not model: every arm runs the same agent with the same model, gpt-5.6-terra. Only the harness around it differs: baseline, baseline-docs, wingfoil.
Against the baseline, baseline-docs is better in 3, worse in 1 and the same in 2 of 6 comparisons across categories C, D, E and F (preliminary: n < 3 in D, E and F).
Against the baseline, wingfoil is better in 2, worse in 1 and the same in 3 of 6 comparisons across categories C, D, E and F (preliminary: n < 3 in D, E and F).
- Step A: the session agent drives WingFoil 0.3.1's workflow, acting in each step's role and stopping at each review gate for the approver; its docs control carries every role's rules.
- Read arm against arm within Codex and one model at a time; gpt-6-luna, the weak model, ran every scenario three times, gpt-5.6-terra S1 three times and the others once.
- The comparison with 13fda3099afa/1 (v0.2's manual on WingFoil v0.3.0) carries that execution's erratum (bug-036): its wingfoil arm's resumed turns ran on another model.
- Approver policy v2 answered the wingfoil arm's 77 approval requests in 78 steps; in the other step the session asked nothing and the next one recovered.
- Codex ran on a ChatGPT Plus subscription; its cost is API-equivalent from a pinned OpenAI price table, not spent.
- Preliminary. The content checks are lexical (bug-026).
A, B and G are not covered in this campaign.
| Category | baseline | baseline-docs | wingfoil wingfoil v0.3.1 · 870bd44 |
|---|---|---|---|
| A — Inception & Specification | not covered in this campaign | ||
| B — Planning & Management | not covered in this campaign | ||
| C — Development S1@1.0 | M-Q1 98.4% n = 3 range 96.4%–99.4% M-K1 0.7278 EUR n = 3 range 0.6958 EUR–0.7504 EUR M-Q1 hold-out 45/48, 48/48, 45/48 n = 3 | M-Q1 99.4% n = 3 range 99.4%–99.4% +1.0 pp ▲ better within variance M-K1 0.8237 EUR n = 3 range 0.7888 EUR–0.8635 EUR +0.0959 EUR ▼ worse beyond variance M-Q1 hold-out 45/48, 46/48, 45/48 n = 3 | M-Q1 99.8% n = 3 range 99.4%–100.0% +1.4 pp ▲ better within variance M-K1 1.1898 EUR n = 3 range 1.0827 EUR–1.2612 EUR +0.4619 EUR ▼ worse beyond variance M-Q1 hold-out 48/48, 46/48, 48/48 n = 3 |
| D — Maintenance & Quality S2@1.0 | M-Q1 95.8% n = 1 preliminary M-D3 1 n = 1 preliminary M-Q1 hold-out 16/17 n = 1 | M-Q1 95.8% n = 1 preliminary ±0.0 pp = same M-D3 1 n = 1 preliminary ±0 = same M-Q1 hold-out 16/17 n = 1 | M-Q1 95.8% n = 1 preliminary ±0.0 pp = same M-D3 1 n = 1 preliminary ±0 = same M-Q1 hold-out 16/17 n = 1 |
| E — Governance & Compliance S8@1.0 | M-E1 10 n = 1 preliminary M-Q1 hold-out 6/7 n = 1 | M-E1 0 n = 1 preliminary −10 ▲ better M-Q1 hold-out 7/7 n = 1 | M-E1 0 n = 1 preliminary −10 ▲ better M-Q1 hold-out 7/7 n = 1 |
| F — Knowledge & Continuity S3@1.0 | M-F1 60.0% n = 1 preliminary M-Q1 hold-out 16/17 n = 1 | M-F1 80.0% n = 1 preliminary +20.0 pp ▲ better M-Q1 hold-out 16/17 n = 1 | M-F1 60.0% n = 1 preliminary ±0.0 pp = same M-Q1 hold-out 16/17 n = 1 |
| G — Collaboration | not covered in this campaign | ||
Each value is the mean of its runs, with its n; a single run is preliminary, and so is a comparison unless both sides have n ≥ 3; a range is shown from n = 3. Deltas are against the baseline. How to read these results; which tools are eligible, and why.
Compared with earlier executions
Across models
Views by profile
- Solo developer (C, D, F)
- Code reviewer (D, E)
- Team developer (C, F, G)
- Tech lead (B, E, G)
- Non-technical manager (A, B)
- Architect (A, E, F)