← WingFoil Benchmark — campaign 81ebb2691f9c, execution 1

Which tools are eligible, and why

A harness gets an arm in this benchmark only when it meets five published criteria. They are applied before an arm is built, and to every tool, WingFoil included: a tool is admitted by the rule, never by choice.

The five criteria

1. agent-and-model — it runs with the campaign's agent and model id: the agent the benchmark meters, on the model every arm shares. A tool that brings its own agent, or forces another model, would compare something else. 2. pinnable — its version can be pinned and installed reproducibly: a release or a commit, installed the same way every time, so that a result names the tool it measured. 3. headless-container — it runs headless in a container: no terminal, browser or person in the loop beyond the approver's answers, which every arm receives from the same neutral approver. 4. workflow-harness — it is a workflow harness, not only a standards or prompt pack: it gives the agent a process to follow, which is what the benchmark measures. 5. no-own-llm — it calls no LLM of its own: every model call is the agent's, so that every arm's cost and tokens are metered the same way.

How a tool is assessed

• One assessment per tool and version, dated, each criterion pass or fail with its evidence and its source. An assessment made from a tool's documentation says so, and is redone when a spike or the arm's own task runs it.
• The verdict follows the criteria: admitted when all five pass, excluded when one fails, with the reason.
• A campaign may pin only a version the register admits. A new release is assessed before a campaign pins it.
• Admitted does not mean measured: which admitted tools get an arm in a release is that release's scope.
• Telemetry is not a criterion: every arm's tool runs with its telemetry off, a deviation published with its setup.

The register

This is the register as it stands when this site was built, so that every execution's site shows the current assessments. Each campaign was checked, when it ran, against the register of that day; an assessment's date is kept in [eligibility/register.yaml](https://github.com/wingfoil/wingfoil-benchmark/blob/main/eligibility/register.yaml).

wingfoil v0.2.2

• agent-and-model: pass — v0.1's reference campaign c82a5e74885b ran it with Claude Code 2.1.280 and the campaign's models (Sonnet 5, Opus 5): the agent works through WingFoil's CLI and MCP server, on the session's model.
• pinnable: pass — The tag v0.2.2 resolves to commit 12537b62; the runner builds it from that commit into a cached tarball and installs the tarball in the run container (REQ-RUN-14, adr-003).
• headless-container: pass — v0.1's campaign and calibration ran it in the run container without a terminal; its gates are answered by the neutral approver (REQ-RUN-17).
• workflow-harness: pass — It drives a process — workflows with phases and approver gates, a Memory of elements with state machines, directives per role — not only a set of rules or prompts.
• no-own-llm: pass — No model call of its own: its CLI and MCP server read and write the repository; every model call is the agent's, metered by the runner.

wingfoil v0.3.0

• agent-and-model: pass — As v0.2.2: the agent works through WingFoil's CLI and MCP server, on the session's model; 0.3.0 changes neither (its CHANGELOG, tag v0.3.0). Assessed from the source and task-095's offline runs with the fake agent; calibration's dry runs are the first real runs on it.
• pinnable: pass — The tag v0.3.0 resolves to commit cc31d7a8 (also published as npm wingfoil@0.3.0 with provenance); the runner builds it from that commit into a cached tarball, as v0.2.2 (REQ-RUN-14).
• headless-container: pass — task-095's docker test sets the arm up in the run container without a terminal (init, the scenario's configuration, the approver member); its gates are answered by the neutral approver (REQ-RUN-17).
• workflow-harness: pass — It drives a process, now with a workflow engine that records and routes each workflow's steps from Memory (workflow start, next, status, finalize, end), beside Memory's state machines and directives per role.
• no-own-llm: pass — No model call of its own. Its new `agent execute` launches an agent CLI through an adapter only for a team.agents entry that declares one: the init scaffold declares none, and the arm's manual keeps the configuration out of the agent's hands, so every model call in a run is the agent's, metered by the runner.

speckit v1.1.0

• agent-and-model: pass — Run in the run container (task-065): `specify init --here --integration claude` installed ten skills under .claude/skills; one Claude Code session on Sonnet 5 followed them (/speckit-specify, -plan, -tasks, -implement) on the session's own model, the only model it reported.
• pinnable: pass — Tag v1.1.0 (commit f1d3a4f8) of github/spec-kit, built into a wheel bundle with its dependencies (14 wheels, each recorded with its digest) and installed from it with `uv tool install --offline --no-index`. PyPI's specify-cli listed 1.0.13 when assessed, behind the tag, so the arm installs from the bundle.
• headless-container: pass — Run in the run container (task-065): install and `init` ran with --network none and no terminal; the session ran its process to the end under `claude -p` without stopping for an answer.
• workflow-harness: pass — A spec-driven process — constitution, specify, plan, tasks, implement — with its templates, scripts and workflows under .specify/; the session wrote a spec, a plan, a data model, contracts and tasks before code.
• no-own-llm: pass — Observed in the run container (task-065): the session never called `specify` or its workflow engine, and its scripts are local shell; install and init ran with --network none. No telemetry or analytics code in its source.

openspec 1.14.0

• agent-and-model: pass — task-070's spike, in the run container: `openspec init --tools claude --profile core --force` wrote six Claude Code skills and six /opsx:* commands; one Claude Code 2.1.280 session on Sonnet 5 followed them (the openspec-propose skill, then openspec-apply-change on resume), on the session's model.
• pinnable: pass — npm @fission-ai/openspec@1.14.0, its tarball matching the registry's integrity, installed with its dependencies into a prefix packed as one artifact, installed and run with --network none (task-070 B1, B2). 1.14.1, published since, is not assessed here.
• headless-container: pass — task-070: init is non-interactive with its flags; the real session ran under `claude -p` with stdin closed, and ended its propose phase asking to go on, a wait the neutral approver answers; its resume started the apply phase and stopped at its budget, part done. The archive phase was not reached: its skill may ask the user to confirm (an approval, when work is incomplete) and archives with mkdir and mv; the CLI's own `archive` takes -y/--yes. No TTY, browser or human-only step.
• workflow-harness: pass — A change-driven process — propose, apply, archive — whose state its CLI checks: the session ran `openspec new change`, `status`, `instructions` and `validate --strict` (task-070 R1).
• no-own-llm: pass — No model or key of its own: the session's model did the work. Its telemetry and its version check against the npm registry are off with OPENSPEC_TELEMETRY=0 (dist/telemetry/opt-out.js), the arm's telemetry_off.

bmad 6.12.1

• agent-and-model: pass — From its documentation (the competitor re-verification of v0.2's planning): `--tools claude-code` installs agents and workflows that Claude Code runs in its own session.
• pinnable: pass — npm bmad-method@6.12.1, installed non-interactively (`npx bmad-method@6.12.1 install … --yes`). Its v7, on main and not released, changes the install.
• headless-container: fail — From its documentation: its planning phases are persona-led facilitations whose questions ask for the project's content — its vision, users, requirements and choices — which only a person holds. The neutral approver answers approvals and clarifications with fixed replies, as for every arm (OpenSpec's questions are of that kind: which change, whether to sync); it cannot supply that content. Only `bmad-dev-auto` and the loop module are documented as unattended, not the process as a whole.
• workflow-harness: pass — A multi-agent method with analysis, planning, solutioning and implementation phases, and their workflows.
• no-own-llm: pass — From its documentation: no model or key of its own; its agents are prompts the configured agent runs.