← How the benchmark works

How the openspec arm is set up, before its agent starts: the openspec harness at the campaign's pin, from the repository's arms/openspec/.

Setup script

#!/usr/bin/env bash
# The openspec arm's setup (REQ-RUN-18, task-071, from task-070's spike): OpenSpec 1.14.0 installed from its artifact and
# initialized as its documentation says for Claude Code. The artifact is the installed tree (`installed.tgz`: the npm
# registry's tarball installed with its dependencies into a prefix), which the runner copies to ~/harness.tgz; HOME and
# WORKSPACE are the runner's (the project rules' snapshot runs this with HOME=/build, WORKSPACE=/build/workspace).
set -euo pipefail
WORKSPACE="${WORKSPACE:-/workspace}"
mkdir -p "$HOME/.local/bin" && tar -xzf "$HOME/harness.tgz" -C "$HOME/.local"
# On the agent's PATH too, not only this shell's: the image's PATH holds $HOME/.local/bin (docker/run-image), and its
# skills call `openspec` directly.
ln -sf "$HOME/.local/openspec/bin/openspec" "$HOME/.local/bin/openspec"
export PATH="$HOME/.local/bin:$PATH"
cd "$WORKSPACE"
# Telemetry off (REQ-FMT-05): the arm's telemetry_off sets it for a run; the project rules' snapshot runs this setup in
# a one-off container that carries no environment, so the setup sets it itself too.
export OPENSPEC_TELEMETRY="${OPENSPEC_TELEMETRY:-0}"
# REQ-RUN-18's flags, verified on 1.14.0: six skills and six /opsx:* commands under .claude/, openspec/config.yaml. The
# scenario's rules are written into that file's context by the runner after this setup (REQ-FMT-14).
openspec init --tools claude --profile core --force

Telemetry

Turned off in the agent's environment:

Operating manual

The operating manual of the openspec arm, given to its agent.

Rules

Written to openspec/config.yaml: the scenario's rules — the directives its developer reads — rendered as the context of OpenSpec's project configuration, after the setup, in place of the file init leaves (its schema kept).

Docs control

openspec-docs: the same agent with the openspec arm's configuration rendered as documentation, without the harness. Kept from the configuration: openspec/.

ContentRenderedWhy
config.yaml's context:yesthe scenario's rules, which the manual sends the agent to read
config.yaml's schema:nothe tool's own mechanics
per-artifact rules: and operations: guidancenoabsent: the rules generator writes the schema and the context only
commands and skills (.claude/commands/opsx, .claude/skills/openspec-*)nothe tool's own mechanics: how its process is followed
accepted specs and changes (openspec/specs, openspec/changes)noempty at setup: the agent's own work, not the project's rules

Git identity

Every arm commits as the same git identity, “Benchmark Approver” (adr-003, decision 7). In the openspec arm, a competitor arm, it carries no approval authority: it is only a name.

Contest

Contest this setup: say which step is not as the tool's official documentation says, and how to correct it. An accepted correction runs as a new campaign; this one stays published.