← campaign 697e79b75e41, execution 1
Comparison: campaign 697e79b75e41, execution 1 against campaign 13fda3099afa, execution 1
Pins that differ
- models.slices: none → gpt-6-luna
- agent.effort.gpt-6-luna: none → medium
- harnesses.wingfoil: wingfoil v0.3.0 → wingfoil v0.3.1
- arm_digests.baseline-docs: 2ac51091da99 → 64e37c838830
- arm_digests.wingfoil: 45a5cce3de19 → cf70ad7ed9aa
Each arm against its own baseline
| Category | Scenario | Arm | Model | Metric | campaign 13fda3099afa, execution 1 | campaign 697e79b75e41, execution 1 | Change |
|---|---|---|---|---|---|---|---|
| C | S1@1.0 | baseline-docs | gpt-5.6-terra | M-Q1 | −0.4 pp n = 3 / 3, range 99.4%–100.0% / 100.0%–100.0% within variance | +1.0 pp n = 3 / 3, range 99.4%–99.4% / 96.4%–99.4% within variance | +1.4 pp |
| C | S1@1.0 | baseline-docs | gpt-5.6-terra | M-K1 | +0.0440 EUR n = 3 / 3, range 0.6621 EUR–0.7858 EUR / 0.6352 EUR–0.7702 EUR within variance | +0.0959 EUR n = 3 / 3, range 0.7888 EUR–0.8635 EUR / 0.6958 EUR–0.7504 EUR beyond variance | +0.0519 EUR |
| C | S1@1.0 | wingfoil | gpt-5.6-terra | M-Q1 | −0.2 pp n = 3 / 3, range 99.4%–100.0% / 100.0%–100.0% within variance | +1.4 pp n = 3 / 3, range 99.4%–100.0% / 96.4%–99.4% within variance | +1.6 pp |
| C | S1@1.0 | wingfoil | gpt-5.6-terra | M-K1 | +0.6237 EUR n = 3 / 3, range 1.1666 EUR–1.4108 EUR / 0.6352 EUR–0.7702 EUR beyond variance | +0.4619 EUR n = 3 / 3, range 1.0827 EUR–1.2612 EUR / 0.6958 EUR–0.7504 EUR beyond variance | −0.1618 EUR |
| D | S2@1.0 | baseline-docs | gpt-5.6-terra | M-Q1 | ±0.0 pp n = 1 / 1 preliminary | ±0.0 pp n = 1 / 1 preliminary | ±0.0 pp |
| D | S2@1.0 | baseline-docs | gpt-5.6-terra | M-D3 | ±0 n = 1 / 1 preliminary | ±0 n = 1 / 1 preliminary | ±0 |
| D | S2@1.0 | wingfoil | gpt-5.6-terra | M-Q1 | ±0.0 pp n = 1 / 1 preliminary | ±0.0 pp n = 1 / 1 preliminary | ±0.0 pp |
| D | S2@1.0 | wingfoil | gpt-5.6-terra | M-D3 | ±0 n = 1 / 1 preliminary | ±0 n = 1 / 1 preliminary | ±0 |
| E | S8@1.0 | baseline-docs | gpt-5.6-terra | M-E1 | −10 n = 1 / 1 preliminary | −10 n = 1 / 1 preliminary | ±0 |
| E | S8@1.0 | wingfoil | gpt-5.6-terra | M-E1 | −10 n = 1 / 1 preliminary | −10 n = 1 / 1 preliminary | ±0 |
| F | S3@1.0 | baseline-docs | gpt-5.6-terra | M-F1 | +20.0 pp n = 1 / 1 preliminary | +20.0 pp n = 1 / 1 preliminary | ±0.0 pp |
| F | S3@1.0 | wingfoil | gpt-5.6-terra | M-F1 | +40.0 pp n = 1 / 1 preliminary | ±0.0 pp n = 1 / 1 preliminary | −40.0 pp |
Not compared
- S1@1.0 baseline (gpt-6-luna): no slice of model gpt-6-luna in 13fda3099afa/1
- S1@1.0 baseline-docs (gpt-6-luna): no slice of model gpt-6-luna in 13fda3099afa/1
- S1@1.0 wingfoil (gpt-6-luna): no slice of model gpt-6-luna in 13fda3099afa/1
- S2@1.0 baseline (gpt-6-luna): no slice of model gpt-6-luna in 13fda3099afa/1
- S2@1.0 baseline-docs (gpt-6-luna): no slice of model gpt-6-luna in 13fda3099afa/1
- S2@1.0 wingfoil (gpt-6-luna): no slice of model gpt-6-luna in 13fda3099afa/1
- S3@1.0 baseline (gpt-6-luna): no slice of model gpt-6-luna in 13fda3099afa/1
- S3@1.0 baseline-docs (gpt-6-luna): no slice of model gpt-6-luna in 13fda3099afa/1
- S3@1.0 wingfoil (gpt-6-luna): no slice of model gpt-6-luna in 13fda3099afa/1
- S8@1.0 baseline (gpt-6-luna): no slice of model gpt-6-luna in 13fda3099afa/1
- S8@1.0 baseline-docs (gpt-6-luna): no slice of model gpt-6-luna in 13fda3099afa/1
- S8@1.0 wingfoil (gpt-6-luna): no slice of model gpt-6-luna in 13fda3099afa/1
On one side only
None.