← WingFoil Benchmark — campaign 697e79b75e41, execution 1
Weak model with the harness against the strong model without it (across models)
Across models: these figures compare different models and are never mixed with the same-model results of the landing page (T14). H-W (dl-032): a weak model with the harness reaches the quality of a strong model without it at a lower cost and in less time. It holds for a scenario when no quality metric is worse beyond the variance and both the cost and the agent time are lower. The weak model of each pair is the one whose baselines cost less in this execution; each scenario is read over the runs both cells have, and below three runs a side it is preliminary: no quality metric can be judged worse there, so H-W rests on cost and time. How it is read.
wingfoil on gpt-6-luna against the baseline on gpt-5.6-terra
H-W holds in 1 and fails in 3 of 4 scenarios; 3 of them preliminary, below three runs a side, where no quality metric can be judged worse.
| Scenario | Runs a side | M-Q1 | Hold-out | Checks | Cost | Agent time | Tokens: fresh · cached · output | H-W |
|---|---|---|---|---|---|---|---|---|
| S1@1.0 | 3 | 99.4 % against 98.4 % better (within variance) | 95.1 % against 95.8 % lower (within variance) | — | 0.07× | 0.44× | 1.74× · 2.02× · 0.90× | holds |
| S2@1.0 | 1 (preliminary) | 95.8 % against 95.8 % same (preliminary) | 94.1 % against 94.1 % same (preliminary) | 0.0 % against 0.0 % same (preliminary) | 0.10× | 1.28× | 1.79× · 2.80× · 1.79× | fails |
| S3@1.0 | 1 (preliminary) | 100.0 % against 91.9 % better (preliminary) | 100.0 % against 94.1 % better (preliminary) | 0.0 % against 0.0 % same (preliminary) | 0.11× | 1.09× | 2.83× · 3.26× · 1.35× | fails |
| S8@1.0 | 1 (preliminary) | 91.3 % against 100.0 % lower (preliminary) | 71.4 % against 85.7 % lower (preliminary) | 100.0 % against 75.0 % better (preliminary) | 0.12× | 1.05× | 2.96× · 3.17× · 1.65× | fails |
Quality: the weak model with the harness against the strong model's baseline, worse only when worse beyond the variance. Cost, time and tokens: the weak harness's figure over the strong baseline's.
Every cell
| Scenario | Cell | n | M-Q1 | Hold-out | Checks | Cost | Agent time | Tokens: processed · output |
|---|---|---|---|---|---|---|---|---|
| S1@1.0 | weak model, harness (wingfoil, gpt-6-luna) | 3 | 99.4 % | 95.1 % | — | 0.0516 € | 5.5 min | 2,640,694 · 25,066 |
| S1@1.0 | strong model, baseline (baseline, gpt-5.6-terra) | 3 | 98.4 % | 95.8 % | — | 0.7278 € | 12.3 min | 1,321,450 · 27,836 |
| S1@1.0 | weak model, baseline (baseline, gpt-6-luna) | 3 | 96.2 % | 93.8 % | — | 0.0128 € | 1.8 min | 465,108 · 9,017 |
| S1@1.0 | weak model, docs control (baseline-docs, gpt-6-luna) | 3 | 86.5 % | 88.9 % | — | 0.0212 € | 2.4 min | 787,374 · 12,930 |
| S1@1.0 | strong model, harness (wingfoil, gpt-5.6-terra) | 3 | 99.8 % | 98.6 % | — | 1.1898 € | 16.3 min | 2,816,063 · 34,024 |
| S2@1.0 | weak model, harness (wingfoil, gpt-6-luna) | 1 | 95.8 % | 94.1 % | 0.0 % | 0.0356 € | 3.2 min | 1,973,143 · 12,922 |
| S2@1.0 | strong model, baseline (baseline, gpt-5.6-terra) | 1 | 95.8 % | 94.1 % | 0.0 % | 0.3451 € | 2.5 min | 734,848 · 7,199 |
| S2@1.0 | weak model, baseline (baseline, gpt-6-luna) | 1 | 95.8 % | 94.1 % | 0.0 % | 0.0139 € | 1.2 min | 514,593 · 5,321 |
| S2@1.0 | weak model, docs control (baseline-docs, gpt-6-luna) | 1 | 95.8 % | 94.1 % | 0.0 % | 0.0162 € | 1.6 min | 745,728 · 7,676 |
| S2@1.0 | strong model, harness (wingfoil, gpt-5.6-terra) | 1 | 95.8 % | 94.1 % | 0.0 % | 0.8028 € | 4.9 min | 2,145,026 · 14,774 |
| S3@1.0 | weak model, harness (wingfoil, gpt-6-luna) | 1 | 100.0 % | 100.0 % | 0.0 % | 0.0528 € | 5.8 min | 2,868,226 · 23,690 |
| S3@1.0 | strong model, baseline (baseline, gpt-5.6-terra) | 1 | 91.9 % | 94.1 % | 0.0 % | 0.4641 € | 5.3 min | 889,555 · 17,534 |
| S3@1.0 | weak model, baseline (baseline, gpt-6-luna) | 1 | 91.9 % | 94.1 % | 0.0 % | 0.0116 € | 1.6 min | 455,343 · 6,914 |
| S3@1.0 | weak model, docs control (baseline-docs, gpt-6-luna) | 1 | 91.9 % | 94.1 % | 0.0 % | 0.0181 € | 2.2 min | 816,567 · 11,302 |
| S3@1.0 | strong model, harness (wingfoil, gpt-5.6-terra) | 1 | 91.9 % | 94.1 % | 0.0 % | 1.0764 € | 11.0 min | 2,488,401 · 27,779 |
| S8@1.0 | weak model, harness (wingfoil, gpt-6-luna) | 1 | 91.3 % | 71.4 % | 100.0 % | 0.0445 € | 5.9 min | 2,033,740 · 22,686 |
| S8@1.0 | strong model, baseline (baseline, gpt-5.6-terra) | 1 | 100.0 % | 85.7 % | 75.0 % | 0.3716 € | 5.6 min | 645,598 · 13,719 |
| S8@1.0 | weak model, baseline (baseline, gpt-6-luna) | 1 | 100.0 % | 100.0 % | 81.3 % | 0.0120 € | 1.3 min | 427,156 · 6,488 |
| S8@1.0 | weak model, docs control (baseline-docs, gpt-6-luna) | 1 | 100.0 % | 100.0 % | 100.0 % | 0.0202 € | 2.2 min | 838,510 · 11,849 |
| S8@1.0 | strong model, harness (wingfoil, gpt-5.6-terra) | 1 | 100.0 % | 100.0 % | 100.0 % | 0.8778 € | 11.2 min | 2,051,392 · 23,297 |
A local model's time, estimated
| Scenario | slow (300 · 15 tokens/s): no cache · prefix cache | mid (1500 · 40 tokens/s): no cache · prefix cache | fast (5000 · 100 tokens/s): no cache · prefix cache | Strong baseline's agent time | Slowest profile that keeps up: no cache · prefix cache |
|---|---|---|---|---|---|
| S1@1.0 | 2.9 h · 39.5 min | 39.8 min · 12.8 min | 13.0 min · 4.9 min | 12.3 min | none · fast |
| S2@1.0 | 2.1 h · 22.8 min | 27.3 min · 7.1 min | 8.7 min · 2.7 min | 2.5 min | none · none |
| S3@1.0 | 3.1 h · 37.9 min | 41.7 min · 12.2 min | 13.5 min · 4.6 min | 5.3 min | none · fast |
| S8@1.0 | 2.3 h · 36.4 min | 32.0 min · 11.7 min | 10.6 min · 4.5 min | 5.6 min | none · fast |
The profiles are assumptions, not measurements: a local model would read every token processed (without a prefix cache) or only the fresh ones (with one) at the prefill speed, and write the output at the decode speed; tool time is left out. They are replaced by measurements once a local model runs.