← WingFoil Benchmark — campaign c82a5e74885b, execution 2

How the benchmark works

This benchmark measures where a harness for AI coding agents makes a difference. A harness is the process and memory tooling around an agent: here, WingFoil. The same agent and model work through the same tasks with and without it, and the results are compared by fixed, published rules. This page states those rules, including the ones that limit what the results can show.

What is compared

The benchmark compares harnesses, not models. In one comparison the agent (Claude Code), its version and the model are the same in every arm. Only the harness and the files that come with it change. Runs on a second model are a separate comparison and are never mixed with same-model results.

The agent may also call a smaller model for its own work, beside the run's model: in v0.1, Claude Code called Haiku 4.5 now and then, for a small share of the cost. The models each step used, with their tokens and cost, are recorded with the run. Their cost is part of the arm's cost; the arm's token counts are the run's model's only.

There are three arms:

• baseline: the agent, the starting project (the seed) and a short operating manual. It is the plain agent and the reference for every comparison.
• baseline-docs: the same as baseline, plus the same information the wingfoil arm gets, written as plain Markdown in one file, PROJECT_RULES.md.
• wingfoil: the same as baseline, plus WingFoil at the campaign's pinned version, set up and configured, with its MCP server, and an operating manual that tells the agent how to use it.

The baseline-docs arm is a control. A harness also gives the agent more context, and more context alone might explain a difference (threat T3). baseline-docs gets the same information as wingfoil, without the harness. Its file is generated by a program, never edited by hand. The program reads the wingfoil arm's configuration as the agent would meet it, captured by running the wingfoil arm's own setup, and writes the same bytes every time. What baseline-docs received is kept with the execution's results. A scenario's project rules exist only in the arms' files: baseline gets none, baseline-docs gets them as Markdown, wingfoil gets them as WingFoil directives and records.

What counts as "the same information": every kind of content in the wingfoil arm's configuration, and whether the generator passes it on to baseline-docs.

Each arm has one fixed operating manual. It is copied into the project as CLAUDE.md, the file the agent reads first. It maps what a step asks for to the harness's commands. A longer manual is also more context, so each manual's size is measured (in tokens, approximated as the file's bytes divided by 4) and reported for every run. The three manuals are published: baseline, baseline-docs, wingfoil.

The wingfoil manual sets the process the agent follows with WingFoil. WingFoil itself does not require this process: the manual chose it, as a disciplined use of WingFoil. Before changing code, the agent reads its role's rules and the approved decisions. While it works, it records each request as a task and each design decision as a decision record, submits them and, when its work needs an approval, asks for it and stops; once the neutral approver replies, it runs the approval itself. That reading, writing and the approval round trips are part of the wingfoil arm's cost, turns and time. A lighter use of WingFoil could cost less, and a stricter one more; v0.1 measures only this one. The results measure WingFoil as this manual uses it, and the manual is published so that the process can be judged with them.

Every step's request (its prompt) is the same text in every arm, and names no harness. A scan checks each scenario for this before it runs: a prompt that names a harness or tool, or a seed or prompt that contains an expected value or a hidden test's name, is refused.

How a run goes

A campaign file fixes everything that can change a result: each harness's version, the scenarios and their versions, the agent and its version, the model, the number of repetitions, the caps, the approver policy's version, and the dollar-to-euro rate used for costs. A campaign must include the baseline arm, so the baseline is rerun every time. This execution's pins are listed under This execution.

A run is one scenario, in one arm, on one model. Each run happens in a fresh container that holds only a copy of the seed and the arm's files. Nothing from the benchmark's own repository, its hidden tests or the private hold-out is in it. The agent has internet access, as it would in real work.

Before the first step, the arm's setup runs. Its time and cost are recorded as setup, apart from the steps. In the wingfoil arm, the setup installs WingFoil built from its pinned commit, with that commit's exact dependency versions.

Each step starts a new agent session, with no memory of earlier steps. Only the project's files carry anything from one step to the next. The agent's own automatic memory is cleared before every step, in every arm. After each step the runner saves a snapshot of the project (a git commit), which is what gets scored.

The agent works without a human. When a session ends by asking a question or asking for approval, a neutral approver answers with a fixed reply, the same for every arm. Each reply is one intervention, and interventions are reported per arm.

• An approval request gets: "Approved. Proceed."
• A question gets: "No further input is available. Make the most reasonable choice, record it, and proceed."
• After 3 interventions in a step, the step ends.

Whether a session is waiting for input is decided by a small, fixed set of written rules, not by a model. First, the last message is checked for approval phrases anywhere in it, such as "are you sure", "please confirm" or "do you want me to". If none matches, the message counts as a question when its last line ends with a question mark. Otherwise the session is not waiting, and the step ends. Code quoted in the message is ignored. The rules have a version, which is part of the approver policy's version. Changing a rule makes a new version and is never done quietly. The list is small and may misread unusual wording. A session that fails is a failed step, not a waiting one, and is never answered.

In the wingfoil arm, WingFoil's process needs a person with the approver role. The arm's configuration declares a member "Benchmark Approver" with that role, and the container's git identity is that member. After the neutral approver's reply, the agent may run WingFoil's approval commands itself. The decision is always the neutral approver's fixed reply; the agent only carries it out.

Every step has a time cap and a token cap, and every run has a cost cap. When a cap is hit, the step or the run ends, and what exists at that moment is scored as it stands. The campaign also stops starting new runs once its total cost reaches its ceiling.

The run's cost cap is passed to the agent, which stops by itself, at most one turn past the cap. A session can therefore spend a little more than the cap. In a test, a session capped at 0.04 USD stopped at 0.0419 USD. The real cost is recorded as reported.

A step killed at its time cap reports no cost. It is counted at its upper bound: the most it was allowed to spend, which is what was left of the run's cost cap. The budget therefore never counts less than may have been spent. Since that bound is everything left of the run's cost cap, the run then ends. The work done before the kill is kept and scored. A step whose session output (the stream the runner reads from the agent) passes the runner's bound of 256 MiB is a failed step: its cost is not reported, so it too is counted at its upper bound, and the run ends there.

The token cap is checked between the agent's invocations, not during one. A step whose tokens exceed the cap is not resumed after its current session. Within one session, only the cost cap bounds it.

When the subscription's rate limit rejects a session (HTTP 429), the step is not failed: the runner waits 2, 5, 15 and then 30 minutes, resuming the same session with the message "Continue." after each wait. The waits do not count against the step's time cap, and they are recorded with the step, apart from the approver's interventions. The four waits are the step's, not each rejection's, and a step over its token cap or with nothing left of the run's cost cap is not resumed. If the limit still holds after the last wait, the step ends as when the subscription's usage limit is reached: the run ends there, and the campaign starts no further run.

The agent's web use is recorded from its web tools, WebFetch and WebSearch. Network use through shell commands (for example curl, or installing packages) is not detected. Reported web use is therefore a lower bound.

Each arm with a harness declares the harness capabilities it offers, each as true or false. A false entry is a known gap, and anything not declared counts as not offered. The baseline arms have no harness and declare nothing. The declaration is reassessed whenever the pinned WingFoil changes, and what this execution's runs declared is listed under This execution.

A scenario can list the capabilities it needs. A run of a harness arm that lacks one of them is marked an expected failure. It is still run and scored, and it is shown as a loss, with the missing capability named. Baseline arms are the reference and are never marked. Among v0.1's scenarios, only S8 lists a capability (rules delivered by role).

What is measured

Scoring happens after the run, outside the run's container. Each snapshot is rebuilt from what the run stored, and checked against what the run recorded. The tests run in a separate container with no network, and the same snapshot always gives the same score.

M-Q1, hidden-test pass rate: the share of hidden tests that pass. Hidden tests are written by the benchmark (or taken from an official conformance suite) and never shown to the agent. Each scenario says which groups of tests (suites) apply after which step. The final snapshot is scored against every suite.

How hidden tests are counted:

• A hidden test is a single test case, named by its file and name. Skipped tests and to-do tests count nowhere.
• The total is fixed in advance: every suite is run once on the seed, and the tests it reports are the total. It is the same for every run of a scenario version.
• On a snapshot, the passed tests are those reported passing. Every other test counts as failed: one that failed, one that ran out of time (120 seconds per test), or one never reported because its file did not load or was stopped.
• A problem in the benchmark's own tests is not a zero. It stops the scoring of that scenario version with an error, and no result is produced.
• A step the run never reached, and the final snapshot of a run that did not finish, have no score: they are "not reached".

Some scenarios also have hold-out tests: extra cases kept in a private repository, so that looking up the public tests online does not help (threat T13). A hold-out result is the number of hold-out tests passed and their total, per suite. It is run apart, with its own total taken on the seed, and stored apart from the public results. Test names and messages are never published. Hold-out tests do not enter M-R1, M-F1 or M-D3.

M-Q2, static quality, of the code the run wrote. Each indicator is reported on its own, and there is no combined score:

• The files measured are the source files (.ts, .tsx, .mts, .cts, .js, .jsx, .mjs, .cjs; not .d.ts declaration files, not node_modules/) that the final snapshot adds or changes from the seed. Every file the arm's setup touched is left out, whatever the harness. With no such file, M-Q2 is "not applicable"; with no final snapshot, "not reached".
• Lint findings per 1,000 lines, from ESLint with the benchmark's fixed configuration, never the project's: the published rule sets @eslint/js recommended and typescript-eslint recommended, without type checking. A file that does not parse is one finding.
• Cyclomatic complexity, read for each function, reported as the mean and the maximum.
• Duplicated lines, as a percentage, from jscpd with a minimum block of 50 tokens.
• Test coverage: the lines of the measured files (tests excluded) that ran while the project's own npm test ran, counted whether its tests pass or fail. With no test script, coverage is 0. The agent's own tests may depend on time or order, so coverage can vary between two scorings of one run.

Some results are read by checks. A check is a file of the scenario's public oracle, with a kind and the steps it is scored at. A check looks for content, never for one harness's files or formats. There are four kinds:

• content: some text was written (see below);
• unchanged: given lines of the seed still stand, in order, in the same file;
• dependencies: runtime dependencies added since the seed;
• ast: rules matched in the TypeScript code's syntax.

The first three read text in the scorer itself. The ast check parses the code in the scoring container. No check runs the agent's code.

A content check looks for groups of words, ignoring case and spacing. It passes at a step when every group matches in one place: the lines the step added to one tracked text file, or one of the step's commit messages. Text that was already there does not count. A check that the step's prompt would satisfy is refused, so an agent cannot pass it by copying the prompt. The check asks only that some record exists, not how good it is or how long. A code comment that states the change counts as a record.

The ast rules match syntax, not meaning, and this has limits. A clock or random function reached through another name (an alias) is not seen. Test files and declaration files are left out. Node's crypto random functions (for example crypto.randomUUID()) count as randomness.

Scenario S8 gives four project rules, published as the wingfoil arm's directives for S8: R1, no new runtime dependency; R2, every exported function has a documentation block; R3, no wall clock or randomness in the domain code; R4, errors are returned as a Result value, never thrown, in the domain code. No prompt mentions them.

M-E1, directive violations: the number of rule violations found by the dependencies and ast checks, per step and per rule, each with where it is. The landing page sums a run's violations over the steps the checks score. Fewer is better.

M-F1, decision consistency: the share of earlier decisions that the final snapshot respects or revises explicitly. Each decision has hidden tests named after it; a decision the scenario revises on purpose also has a content check that looks for a record of the revision. A revision the scenario asks for counts only when it is recorded: a silent change is a failure.

M-F2, next-change cost: the cost and the M-Q1 of each step after the first. It is a reading of those steps, which are built on the code earlier steps left. No computation separates the effect of the earlier code.

M-D3, regressions: the public hidden tests that passed on the seed and fail on the final snapshot. The seed's own results are measured, not assumed.

M-K1, cost: input, output and cache tokens, and the cost in euro as if paid at API prices, whatever the actual billing. M-K2: wall time, turns and interventions.

M-K3, setup cost: the setup's usage, cost and wall time, and the size of the operating manual in tokens. In v0.1 no setup runs the agent, so its cost is 0.

M-K4, break-even: how many steps it takes for an arm's setup cost to pay for itself through cheaper steps. Each arm other than the baseline, baseline-docs included, is paired with the baseline of the same scenario version and model. The rules are applied in this order:

• "not applicable" when the arm's mean final M-Q1 is lower than the baseline's (a final not reached counts as 0);
• otherwise "never" when the arm's mean step cost is not lower (a killed step counted at its bound);
• otherwise the arm's mean setup cost divided by the difference in mean step cost.

An expected failure is computed from what it measured. In v0.1 a number is always 0, or one of the two special cases: the setup costs nothing, and the harness's overhead is in each step's cost, since every new session reads the manual again.

Determinism (M-R1 to M-R3) compares repetitions of the same arm, scenario and pins. In v0.1 only S1 has repetitions (three).

M-R1, outcome agreement: the share of the final snapshot's public hidden tests that get the same verdict in every repetition. Hold-out tests do not enter it.

M-R2, interface similarity: how alike the repetitions' public interfaces are. The interface is read from the code's syntax, with no type checker, so code that does not compile still has one. It is read from the TypeScript sources among M-R3's paths, not declaration files and not tests. Each exported declaration is one entry: its file name and its signature, without bodies, initial values or comments. A file that does not parse is one entry saying so. The same signature in another file is a different entry.

M-R3, structure similarity: how alike the repetitions' sets of file paths are. The paths are every file of the final snapshot, less every path the setup touched, and less generated paths: anything under node_modules/, dist/, build/ or coverage/, *.tsbuildinfo, and lockfiles. The seed's files are common to all repetitions, so they raise the similarity.

M-R2 and M-R3 are the mean, over each pair of repetitions, of the shared entries divided by all entries (Jaccard similarity). Only repetitions that reached their final snapshot are compared; the others are listed and left out, and n counts the runs compared. With fewer than two, there is no value. The pins each run records for itself (the scenario's content hash, the harness commit, the scoring tools) must match; when one differs there is no value, and the differing pins are named. No threshold for "equivalent" is applied: the first campaign provides reference values.

Harness files written during the steps (for example WingFoil's records under .wingfoil/memory/) count in M-R3. Only the setup's files are left out, whatever the harness.

A run whose final snapshot was not reached is a loss. Its final M-Q1 counts as 0 of the total. For M-F1 no decision is consistent, and for M-D3 every test that passed on the seed counts as regressed. A single step not reached is left out of that step's values and is not a loss by itself.

How to read the results

Each covered category is read through fixed metrics, from its main scenario:

A category is covered when one of the execution's scenarios has it as its main category. Categories A, B and G have no scenario in v0.1 and are listed as "not covered", with the release that plans them.

M-D1 (defects fixed) and M-D2 (false or duplicate reports rejected) are not reported on their own in v0.1. S2's defect tests are counted in M-Q1, and each of S2's suites has its own tally on category D's page.

Each arm other than the baseline is compared with the baseline of the same scenario version and model, metric by metric. A value is the mean of its runs, shown with its n and, from 3 runs, its range (the chart's whisker shows it from 2). It is better or worse when the means differ, by the metric's direction (higher is better for M-Q1 and M-F1; lower for M-K1, M-D3 and M-E1), and the same when they are equal. A metric one side lacks reads "not measured" and is no comparison.

A difference is beyond variance only when both sides have at least 3 runs and their ranges (lowest to highest) do not overlap. Otherwise it is within variance. No claim of statistical significance is made in v0.1.

A value from a single run is labelled preliminary and has no range. A comparison where either side is a single run is preliminary too, and says so even when the other side has more runs. The v0.1 reference campaign repeats only S1, so its categories D, E and F are preliminary.

M-E1 is not comparable for an arm when one of its runs did not reach a step the rule checks score, or was not scored by them. Such an arm gets no outcome and no count in the headline. When it is the baseline, each other arm reads "no comparison". A run that stopped early would otherwise look better, with fewer steps to break rules in.

The headline is generated from the comparisons, with no hand-written text. It has one sentence per arm, of the form "Against the baseline, arm is better in b, worse in w and the same in s of n comparisons across categories …", then the categories where a side is a single run, marked preliminary, then the categories not covered. A difference at n = 1 counts, marked preliminary.

The landing page's one chart shows M-Q1 on the final snapshot for every covered scenario, one bar per arm, with its value, its n and, from two runs, its range. A legend names the arms, and each is told apart by hatching as well as colour.

Better, worse and the same use the same markup and weight: a word and a symbol (▲ better, ▼ worse, = same), never colour alone. Arms are listed baseline first, then by name.

Every value is published with the list of runs it was computed from and its n. A value that compares two groups, such as the break-even, carries both groups' runs. Each run can be inspected with the bench run show command named beside it.

A loss is never skipped or hidden. A run counts as a loss when it did not reach its final snapshot, or when it is an expected failure. Each loss is shown with its reason, and an expected failure with the missing capability.

When the hold-out was not scored (for example, not configured), the hold-out reads "not scored", never 0. Scored hold-out results are marked "hold-out", with their n, beside the public ones.

S2's answer key (where each injected defect is, and what it is) is kept only in the private hold-out and is never published. The public scenario describes the kinds of defect only. The site never reads the hidden tests' files, the hold-out or the agents' transcripts.

Results from a second model (the Opus 5 comparison slice) are reported apart from the main model's, never mixed with them. In v0.1 the slice is one run of S1 in the wingfoil arm only, because a slice in every arm would have cost too much for v0.1's budget. It is read against the same arm's runs on the main model: it shows how the wingfoil arm fares on Opus 5, not how the harness's effect changes with the model.

A finding note is a Markdown file the maintainer generates from one execution, for one scenario, metric and set of arms. It holds fixed sections: the campaign, the WingFoil commit, the scenario and version, the runs, the metric values, and links to each run. Missing values are stated, never invented. It ends with a section shaped as a WingFoil bug or decision record, with the facts filled in and the judgement left to the maintainer. The note is written only into the benchmark's repository; the maintainer files it in WingFoil by hand. Finding notes are not on this site.

S1's hidden tests run on third-party material: the json-patch-tests suite (Apache-2.0) and examples from RFC 6901 and RFC 7386 (BSD-3-Clause, IETF Trust). Their sources, pins and licences are in the notice, with the Apache-2.0 and BSD-3-Clause (IETF) texts.

What could be wrong

These threats were listed before any run, each with what is done about it.

• T1, maintainer bias. The benchmark is designed by WingFoil's maintainer. Mitigation: the questions and metrics were fixed before any run; losses are published like wins; metrics are tool-neutral; some scenarios are ones where competitors are expected to be strong; setups are public, with a contest process.
• T2, category bias. WingFoil's unique features sit in category E. Mitigation: E is one category of seven, with no overall score; the governance metrics measure outcomes, not file formats.
• T3, more context, not the harness. Mitigation: the baseline-docs control, generated by a program; each manual's size reported.
• T4, prompt leakage or asymmetry. Mitigation: identical prompts, a leak scan, and published operating manuals.
• T5, training contamination. Public seeds and specifications may be known to the model. Mitigation: hold-out tests; fictional domains (a later scenario, S4); conformance suites used as tests, never as seeds.
• T6, the model is not deterministic. Mitigation: repetitions where affordable; n on every number; preliminary labels.
• T7, model or agent change over time. Mitigation: the model and agent version are pinned; the baseline is rerun in every campaign; across campaigns only differences are compared.
• T8, few runs, because of the budget. Mitigation: preliminary labels; no claim of significance in v0.1.
• T9, the approver policy favours some tools, for example tools that ask more questions. Mitigation: one versioned policy for every arm; interventions reported per arm.
• T10, gaps in the harness. At design time, the WingFoil under development (v0.2 pre-release, commit 3df305e) had no workflow engine; its workflows were followed by hand. Mitigation: expected failures, published as losses; the public campaign runs on the latest released WingFoil, v0.2.2, which has no workflow engine either.
• T11, the tests capture the specification only partly. Mitigation: several metrics per goal; a rubric judge from v0.3.
• T12, unequal setup effort across tools. Mitigation: setup from official documentation only, scripted and published, with deviations published.
• T13, looking up answers. With internet access, an agent may fetch public material that overlaps the tests, for example S1's official conformance suite. Mitigation: web requests recorded and reported per arm (shell network use is not seen); hold-out tests added to public ones; fictional domains (a later scenario, S4) where lookup cannot help. Access is the same for every arm.
• T14, the harness's effect may depend on the model. Mitigation: a comparison slice on a second model (Opus 5), shown as a separate comparison, never mixed with same-model results. In v0.1 it covers the wingfoil arm only, because a slice in every arm would have cost too much for v0.1's budget, so v0.1 does not yet test whether the harness's effect depends on the model.

This execution

The facts below are not written by hand. They are read from this execution's own files: its campaign file, its runs' records and its aggregated results.

Pins

Budget

Spending

18 aggregated runs of claude-sonnet-5 cost 38.1187 EUR in all. 1 aggregated run of other models, reported apart, cost 23.2645 EUR. Setup costs (M-K3) are not included.

Published material

The operating manuals, S8's rules and the licences of S1's third-party material are published as pages of their own, as the runs used them. S2's answer key and the hold-out's content are never published.

• The directives of S8
• Licence: apache-2.0
• Licence: bsd-3-clause-ietf
• Operating manual of the baseline-docs arm
• Operating manual of the baseline arm
• Operating manual of the wingfoil arm
• Third-party material of S1