← WingFoil Benchmark — campaign 33b54dd89e8b, execution 1

Weak model with the harness against the strong model without it (across models)

Across models: these figures compare different models and are never mixed with the same-model results of the landing page (T14). H-W (dl-032): a weak model with the harness reaches the quality of a strong model without it at a lower cost and in less time. It holds for a scenario when no quality metric is worse beyond the variance and both the cost and the agent time are lower. The weak model of each pair is the one whose baselines cost less in this execution; each scenario is read over the runs both cells have, and below three runs a side it is preliminary: no quality metric can be judged worse there, so H-W rests on cost and time. How it is read.

wingfoil on claude-haiku-5-5 against the baseline on claude-sonnet-5-5

H-W holds in 0 and fails in 5 of 5 scenarios.

ScenarioRuns a sideM-Q1Hold-outChecksCostAgent timeTokens: fresh · cached · outputH-W
S1@1.0399.6 % against 99.8 %
lower (within variance)
95.8 % against 97.9 %
lower (within variance)
—0.44×5.30×4.70× · 16.12× · 6.98×fails
S2@1.03100.0 % against 98.6 %
better (within variance)
100.0 % against 98.0 %
better (within variance)
66.7 % against 44.4 %
better (within variance)
0.29×5.01×2.58× · 7.01× · 6.38×fails
S3@1.03100.0 % against 96.4 %
better (within variance)
100.0 % against 96.1 %
better (within variance)
100.0 % against 0.0 %
better (beyond variance)
0.43×6.27×4.33× · 14.09× · 7.09×fails
S8@1.03100.0 % against 100.0 %
same (within variance)
95.2 % against 100.0 %
lower (within variance)
100.0 % against 75.0 %
better (beyond variance)
0.36×5.49×3.45× · 12.70× · 6.18×fails
S10@1.0378.6 % against 88.1 %
lower (within variance)
75.0 % against 91.7 %
lower (within variance)
69.7 % against 37.5 %
better (within variance)
0.27×3.80×2.92× · 6.75× · 4.66×fails

Quality: the weak model with the harness against the strong model's baseline, worse only when worse beyond the variance. Cost, time and tokens: the weak harness's figure over the strong baseline's.

Every cell

ScenarioCellnM-Q1Hold-outChecksCostAgent timeTokens: processed · output
S1@1.0weak model, harness (wingfoil, claude-haiku-5-5)399.6 %95.8 %—0.1455 €13.4 min6,632,378 · 134,554
S1@1.0strong model, baseline (baseline, claude-sonnet-5-5)399.8 %97.9 %—0.3294 €2.5 min450,172 · 19,285
S1@1.0weak model, baseline (baseline, claude-haiku-5-5)3100.0 %100.0 %—0.0569 €5.7 min1,279,533 · 72,047
S1@1.0weak model, docs control (baseline-docs, claude-haiku-5-5)3100.0 %100.0 %—0.1004 €10.0 min3,036,630 · 118,597
S1@1.0strong model, harness (wingfoil, claude-sonnet-5-5)399.4 %93.8 %—0.6468 €5.0 min1,883,294 · 28,291
S2@1.0weak model, harness (wingfoil, claude-haiku-5-5)3100.0 %100.0 %66.7 %0.0560 €4.6 min2,977,797 · 37,474
S2@1.0strong model, baseline (baseline, claude-sonnet-5-5)398.6 %98.0 %44.4 %0.1958 €0.9 min455,125 · 5,873
S2@1.0weak model, baseline (baseline, claude-haiku-5-5)3100.0 %100.0 %66.7 %0.0235 €1.8 min902,724 · 16,437
S2@1.0weak model, docs control (baseline-docs, claude-haiku-5-5)3100.0 %100.0 %66.7 %0.0320 €2.4 min1,247,145 · 23,850
S2@1.0strong model, harness (wingfoil, claude-sonnet-5-5)3100.0 %100.0 %66.7 %0.3392 €1.7 min1,179,385 · 9,414
S3@1.0weak model, harness (wingfoil, claude-haiku-5-5)3100.0 %100.0 %100.0 %0.1053 €9.5 min5,152,354 · 84,923
S3@1.0strong model, baseline (baseline, claude-sonnet-5-5)396.4 %96.1 %0.0 %0.2459 €1.5 min399,208 · 11,979
S3@1.0weak model, baseline (baseline, claude-haiku-5-5)393.7 %94.1 %0.0 %0.0386 €3.6 min1,062,691 · 41,916
S3@1.0weak model, docs control (baseline-docs, claude-haiku-5-5)395.5 %96.1 %0.0 %0.0616 €5.4 min1,966,226 · 63,073
S3@1.0strong model, harness (wingfoil, claude-sonnet-5-5)3100.0 %96.1 %0.0 %0.5713 €3.4 min1,840,586 · 21,386
S8@1.0weak model, harness (wingfoil, claude-haiku-5-5)3100.0 %95.2 %100.0 %0.0816 €7.4 min3,961,716 · 65,088
S8@1.0strong model, baseline (baseline, claude-sonnet-5-5)3100.0 %100.0 %75.0 %0.2293 €1.3 min347,442 · 10,537
S8@1.0weak model, baseline (baseline, claude-haiku-5-5)3100.0 %95.2 %75.0 %0.0263 €2.4 min756,643 · 26,846
S8@1.0weak model, docs control (baseline-docs, claude-haiku-5-5)3100.0 %95.2 %100.0 %0.0470 €4.4 min1,599,709 · 46,529
S8@1.0strong model, harness (wingfoil, claude-sonnet-5-5)3100.0 %95.2 %100.0 %0.4496 €2.6 min1,371,816 · 16,504
S10@1.0weak model, harness (wingfoil, claude-haiku-5-5)378.6 %75.0 %69.7 %0.0756 €6.8 min3,614,532 · 61,355
S10@1.0strong model, baseline (baseline, claude-sonnet-5-5)388.1 %91.7 %37.5 %0.2819 €1.8 min565,532 · 13,166
S10@1.0weak model, baseline (baseline, claude-haiku-5-5)378.6 %75.0 %45.8 %0.0379 €3.4 min1,341,496 · 36,817
S10@1.0weak model, docs control (baseline-docs, claude-haiku-5-5)378.6 %75.0 %63.6 %0.0595 €5.5 min2,385,639 · 54,889
S10@1.0strong model, harness (wingfoil, claude-sonnet-5-5)392.9 %91.7 %93.9 %0.4277 €2.6 min1,340,668 · 16,487

A local model's time, estimated

Scenarioslow (300 · 15 tokens/s): no cache · prefix cachemid (1500 · 40 tokens/s): no cache · prefix cachefast (5000 · 100 tokens/s): no cache · prefix cacheStrong baseline's agent timeSlowest profile that keeps up: no cache · prefix cache
S1@1.08.6 h · 2.7 h2.2 h · 58.9 min44.5 min · 23.3 min2.5 minnone · none
S2@1.03.5 h · 48.5 min48.7 min · 17.0 min16.2 min · 6.7 min0.9 minnone · none
S3@1.06.3 h · 1.8 h1.5 h · 37.7 min31.3 min · 14.9 min1.5 minnone · none
S8@1.04.9 h · 1.4 h1.2 h · 29.0 min24.1 min · 11.4 min1.3 minnone · none
S10@1.04.5 h · 1.3 h1.1 h · 27.3 min22.3 min · 10.7 min1.8 minnone · none

The profiles are assumptions, not measurements: a local model would read every token processed (without a prefix cache) or only the fresh ones (with one) at the prefill speed, and write the output at the decode speed; tool time is left out. They are replaced by measurements once a local model runs.