← WingFoil Benchmark — campaign 697e79b75e41, execution 1

Weak model with the harness against the strong model without it (across models)

Across models: these figures compare different models and are never mixed with the same-model results of the landing page (T14). H-W (dl-032): a weak model with the harness reaches the quality of a strong model without it at a lower cost and in less time. It holds for a scenario when no quality metric is worse beyond the variance and both the cost and the agent time are lower. The weak model of each pair is the one whose baselines cost less in this execution; each scenario is read over the runs both cells have, and below three runs a side it is preliminary: no quality metric can be judged worse there, so H-W rests on cost and time. How it is read.

wingfoil on gpt-6-luna against the baseline on gpt-5.6-terra

H-W holds in 1 and fails in 3 of 4 scenarios; 3 of them preliminary, below three runs a side, where no quality metric can be judged worse.

ScenarioRuns a sideM-Q1Hold-outChecksCostAgent timeTokens: fresh · cached · outputH-W
S1@1.0399.4 % against 98.4 %
better (within variance)
95.1 % against 95.8 %
lower (within variance)
—0.07×0.44×1.74× · 2.02× · 0.90×holds
S2@1.01 (preliminary)95.8 % against 95.8 %
same (preliminary)
94.1 % against 94.1 %
same (preliminary)
0.0 % against 0.0 %
same (preliminary)
0.10×1.28×1.79× · 2.80× · 1.79×fails
S3@1.01 (preliminary)100.0 % against 91.9 %
better (preliminary)
100.0 % against 94.1 %
better (preliminary)
0.0 % against 0.0 %
same (preliminary)
0.11×1.09×2.83× · 3.26× · 1.35×fails
S8@1.01 (preliminary)91.3 % against 100.0 %
lower (preliminary)
71.4 % against 85.7 %
lower (preliminary)
100.0 % against 75.0 %
better (preliminary)
0.12×1.05×2.96× · 3.17× · 1.65×fails

Quality: the weak model with the harness against the strong model's baseline, worse only when worse beyond the variance. Cost, time and tokens: the weak harness's figure over the strong baseline's.

Every cell

ScenarioCellnM-Q1Hold-outChecksCostAgent timeTokens: processed · output
S1@1.0weak model, harness (wingfoil, gpt-6-luna)399.4 %95.1 %—0.0516 €5.5 min2,640,694 · 25,066
S1@1.0strong model, baseline (baseline, gpt-5.6-terra)398.4 %95.8 %—0.7278 €12.3 min1,321,450 · 27,836
S1@1.0weak model, baseline (baseline, gpt-6-luna)396.2 %93.8 %—0.0128 €1.8 min465,108 · 9,017
S1@1.0weak model, docs control (baseline-docs, gpt-6-luna)386.5 %88.9 %—0.0212 €2.4 min787,374 · 12,930
S1@1.0strong model, harness (wingfoil, gpt-5.6-terra)399.8 %98.6 %—1.1898 €16.3 min2,816,063 · 34,024
S2@1.0weak model, harness (wingfoil, gpt-6-luna)195.8 %94.1 %0.0 %0.0356 €3.2 min1,973,143 · 12,922
S2@1.0strong model, baseline (baseline, gpt-5.6-terra)195.8 %94.1 %0.0 %0.3451 €2.5 min734,848 · 7,199
S2@1.0weak model, baseline (baseline, gpt-6-luna)195.8 %94.1 %0.0 %0.0139 €1.2 min514,593 · 5,321
S2@1.0weak model, docs control (baseline-docs, gpt-6-luna)195.8 %94.1 %0.0 %0.0162 €1.6 min745,728 · 7,676
S2@1.0strong model, harness (wingfoil, gpt-5.6-terra)195.8 %94.1 %0.0 %0.8028 €4.9 min2,145,026 · 14,774
S3@1.0weak model, harness (wingfoil, gpt-6-luna)1100.0 %100.0 %0.0 %0.0528 €5.8 min2,868,226 · 23,690
S3@1.0strong model, baseline (baseline, gpt-5.6-terra)191.9 %94.1 %0.0 %0.4641 €5.3 min889,555 · 17,534
S3@1.0weak model, baseline (baseline, gpt-6-luna)191.9 %94.1 %0.0 %0.0116 €1.6 min455,343 · 6,914
S3@1.0weak model, docs control (baseline-docs, gpt-6-luna)191.9 %94.1 %0.0 %0.0181 €2.2 min816,567 · 11,302
S3@1.0strong model, harness (wingfoil, gpt-5.6-terra)191.9 %94.1 %0.0 %1.0764 €11.0 min2,488,401 · 27,779
S8@1.0weak model, harness (wingfoil, gpt-6-luna)191.3 %71.4 %100.0 %0.0445 €5.9 min2,033,740 · 22,686
S8@1.0strong model, baseline (baseline, gpt-5.6-terra)1100.0 %85.7 %75.0 %0.3716 €5.6 min645,598 · 13,719
S8@1.0weak model, baseline (baseline, gpt-6-luna)1100.0 %100.0 %81.3 %0.0120 €1.3 min427,156 · 6,488
S8@1.0weak model, docs control (baseline-docs, gpt-6-luna)1100.0 %100.0 %100.0 %0.0202 €2.2 min838,510 · 11,849
S8@1.0strong model, harness (wingfoil, gpt-5.6-terra)1100.0 %100.0 %100.0 %0.8778 €11.2 min2,051,392 · 23,297

A local model's time, estimated

Scenarioslow (300 · 15 tokens/s): no cache · prefix cachemid (1500 · 40 tokens/s): no cache · prefix cachefast (5000 · 100 tokens/s): no cache · prefix cacheStrong baseline's agent timeSlowest profile that keeps up: no cache · prefix cache
S1@1.02.9 h · 39.5 min39.8 min · 12.8 min13.0 min · 4.9 min12.3 minnone · fast
S2@1.02.1 h · 22.8 min27.3 min · 7.1 min8.7 min · 2.7 min2.5 minnone · none
S3@1.03.1 h · 37.9 min41.7 min · 12.2 min13.5 min · 4.6 min5.3 minnone · fast
S8@1.02.3 h · 36.4 min32.0 min · 11.7 min10.6 min · 4.5 min5.6 minnone · fast

The profiles are assumptions, not measurements: a local model would read every token processed (without a prefix cache) or only the fresh ones (with one) at the prefill speed, and write the output at the decode speed; tool time is left out. They are replaced by measurements once a local model runs.