← WingFoil Benchmark — campaign 33b54dd89e8b, execution 1
Weak model with the harness against the strong model without it (across models)
Across models: these figures compare different models and are never mixed with the same-model results of the landing page (T14). H-W (dl-032): a weak model with the harness reaches the quality of a strong model without it at a lower cost and in less time. It holds for a scenario when no quality metric is worse beyond the variance and both the cost and the agent time are lower. The weak model of each pair is the one whose baselines cost less in this execution; each scenario is read over the runs both cells have, and below three runs a side it is preliminary: no quality metric can be judged worse there, so H-W rests on cost and time. How it is read.
wingfoil on claude-haiku-5-5 against the baseline on claude-sonnet-5-5
H-W holds in 0 and fails in 5 of 5 scenarios.
| Scenario | Runs a side | M-Q1 | Hold-out | Checks | Cost | Agent time | Tokens: fresh · cached · output | H-W |
|---|---|---|---|---|---|---|---|---|
| S1@1.0 | 3 | 99.6 % against 99.8 % lower (within variance) | 95.8 % against 97.9 % lower (within variance) | — | 0.44× | 5.30× | 4.70× · 16.12× · 6.98× | fails |
| S2@1.0 | 3 | 100.0 % against 98.6 % better (within variance) | 100.0 % against 98.0 % better (within variance) | 66.7 % against 44.4 % better (within variance) | 0.29× | 5.01× | 2.58× · 7.01× · 6.38× | fails |
| S3@1.0 | 3 | 100.0 % against 96.4 % better (within variance) | 100.0 % against 96.1 % better (within variance) | 100.0 % against 0.0 % better (beyond variance) | 0.43× | 6.27× | 4.33× · 14.09× · 7.09× | fails |
| S8@1.0 | 3 | 100.0 % against 100.0 % same (within variance) | 95.2 % against 100.0 % lower (within variance) | 100.0 % against 75.0 % better (beyond variance) | 0.36× | 5.49× | 3.45× · 12.70× · 6.18× | fails |
| S10@1.0 | 3 | 78.6 % against 88.1 % lower (within variance) | 75.0 % against 91.7 % lower (within variance) | 69.7 % against 37.5 % better (within variance) | 0.27× | 3.80× | 2.92× · 6.75× · 4.66× | fails |
Quality: the weak model with the harness against the strong model's baseline, worse only when worse beyond the variance. Cost, time and tokens: the weak harness's figure over the strong baseline's.
Every cell
| Scenario | Cell | n | M-Q1 | Hold-out | Checks | Cost | Agent time | Tokens: processed · output |
|---|---|---|---|---|---|---|---|---|
| S1@1.0 | weak model, harness (wingfoil, claude-haiku-5-5) | 3 | 99.6 % | 95.8 % | — | 0.1455 € | 13.4 min | 6,632,378 · 134,554 |
| S1@1.0 | strong model, baseline (baseline, claude-sonnet-5-5) | 3 | 99.8 % | 97.9 % | — | 0.3294 € | 2.5 min | 450,172 · 19,285 |
| S1@1.0 | weak model, baseline (baseline, claude-haiku-5-5) | 3 | 100.0 % | 100.0 % | — | 0.0569 € | 5.7 min | 1,279,533 · 72,047 |
| S1@1.0 | weak model, docs control (baseline-docs, claude-haiku-5-5) | 3 | 100.0 % | 100.0 % | — | 0.1004 € | 10.0 min | 3,036,630 · 118,597 |
| S1@1.0 | strong model, harness (wingfoil, claude-sonnet-5-5) | 3 | 99.4 % | 93.8 % | — | 0.6468 € | 5.0 min | 1,883,294 · 28,291 |
| S2@1.0 | weak model, harness (wingfoil, claude-haiku-5-5) | 3 | 100.0 % | 100.0 % | 66.7 % | 0.0560 € | 4.6 min | 2,977,797 · 37,474 |
| S2@1.0 | strong model, baseline (baseline, claude-sonnet-5-5) | 3 | 98.6 % | 98.0 % | 44.4 % | 0.1958 € | 0.9 min | 455,125 · 5,873 |
| S2@1.0 | weak model, baseline (baseline, claude-haiku-5-5) | 3 | 100.0 % | 100.0 % | 66.7 % | 0.0235 € | 1.8 min | 902,724 · 16,437 |
| S2@1.0 | weak model, docs control (baseline-docs, claude-haiku-5-5) | 3 | 100.0 % | 100.0 % | 66.7 % | 0.0320 € | 2.4 min | 1,247,145 · 23,850 |
| S2@1.0 | strong model, harness (wingfoil, claude-sonnet-5-5) | 3 | 100.0 % | 100.0 % | 66.7 % | 0.3392 € | 1.7 min | 1,179,385 · 9,414 |
| S3@1.0 | weak model, harness (wingfoil, claude-haiku-5-5) | 3 | 100.0 % | 100.0 % | 100.0 % | 0.1053 € | 9.5 min | 5,152,354 · 84,923 |
| S3@1.0 | strong model, baseline (baseline, claude-sonnet-5-5) | 3 | 96.4 % | 96.1 % | 0.0 % | 0.2459 € | 1.5 min | 399,208 · 11,979 |
| S3@1.0 | weak model, baseline (baseline, claude-haiku-5-5) | 3 | 93.7 % | 94.1 % | 0.0 % | 0.0386 € | 3.6 min | 1,062,691 · 41,916 |
| S3@1.0 | weak model, docs control (baseline-docs, claude-haiku-5-5) | 3 | 95.5 % | 96.1 % | 0.0 % | 0.0616 € | 5.4 min | 1,966,226 · 63,073 |
| S3@1.0 | strong model, harness (wingfoil, claude-sonnet-5-5) | 3 | 100.0 % | 96.1 % | 0.0 % | 0.5713 € | 3.4 min | 1,840,586 · 21,386 |
| S8@1.0 | weak model, harness (wingfoil, claude-haiku-5-5) | 3 | 100.0 % | 95.2 % | 100.0 % | 0.0816 € | 7.4 min | 3,961,716 · 65,088 |
| S8@1.0 | strong model, baseline (baseline, claude-sonnet-5-5) | 3 | 100.0 % | 100.0 % | 75.0 % | 0.2293 € | 1.3 min | 347,442 · 10,537 |
| S8@1.0 | weak model, baseline (baseline, claude-haiku-5-5) | 3 | 100.0 % | 95.2 % | 75.0 % | 0.0263 € | 2.4 min | 756,643 · 26,846 |
| S8@1.0 | weak model, docs control (baseline-docs, claude-haiku-5-5) | 3 | 100.0 % | 95.2 % | 100.0 % | 0.0470 € | 4.4 min | 1,599,709 · 46,529 |
| S8@1.0 | strong model, harness (wingfoil, claude-sonnet-5-5) | 3 | 100.0 % | 95.2 % | 100.0 % | 0.4496 € | 2.6 min | 1,371,816 · 16,504 |
| S10@1.0 | weak model, harness (wingfoil, claude-haiku-5-5) | 3 | 78.6 % | 75.0 % | 69.7 % | 0.0756 € | 6.8 min | 3,614,532 · 61,355 |
| S10@1.0 | strong model, baseline (baseline, claude-sonnet-5-5) | 3 | 88.1 % | 91.7 % | 37.5 % | 0.2819 € | 1.8 min | 565,532 · 13,166 |
| S10@1.0 | weak model, baseline (baseline, claude-haiku-5-5) | 3 | 78.6 % | 75.0 % | 45.8 % | 0.0379 € | 3.4 min | 1,341,496 · 36,817 |
| S10@1.0 | weak model, docs control (baseline-docs, claude-haiku-5-5) | 3 | 78.6 % | 75.0 % | 63.6 % | 0.0595 € | 5.5 min | 2,385,639 · 54,889 |
| S10@1.0 | strong model, harness (wingfoil, claude-sonnet-5-5) | 3 | 92.9 % | 91.7 % | 93.9 % | 0.4277 € | 2.6 min | 1,340,668 · 16,487 |
A local model's time, estimated
| Scenario | slow (300 · 15 tokens/s): no cache · prefix cache | mid (1500 · 40 tokens/s): no cache · prefix cache | fast (5000 · 100 tokens/s): no cache · prefix cache | Strong baseline's agent time | Slowest profile that keeps up: no cache · prefix cache |
|---|---|---|---|---|---|
| S1@1.0 | 8.6 h · 2.7 h | 2.2 h · 58.9 min | 44.5 min · 23.3 min | 2.5 min | none · none |
| S2@1.0 | 3.5 h · 48.5 min | 48.7 min · 17.0 min | 16.2 min · 6.7 min | 0.9 min | none · none |
| S3@1.0 | 6.3 h · 1.8 h | 1.5 h · 37.7 min | 31.3 min · 14.9 min | 1.5 min | none · none |
| S8@1.0 | 4.9 h · 1.4 h | 1.2 h · 29.0 min | 24.1 min · 11.4 min | 1.3 min | none · none |
| S10@1.0 | 4.5 h · 1.3 h | 1.1 h · 27.3 min | 22.3 min · 10.7 min | 1.8 min | none · none |
The profiles are assumptions, not measurements: a local model would read every token processed (without a prefix cache) or only the fresh ones (with one) at the prefill speed, and write the output at the decode speed; tool time is left out. They are replaced by measurements once a local model runs.