diff --git a/docs/movement_campaign.md b/docs/movement_campaign.md index 492ef78..7638b09 100644 --- a/docs/movement_campaign.md +++ b/docs/movement_campaign.md @@ -1474,3 +1474,79 @@ PRIMARY metrics are dmg/run and wins/run; hit rate is never the verdict. The pre Reference `tfil`: 115.3 dmg/run, 1.21 wins/run, 17.40% incoming, 384 px. Highest wins delta: `wide_spread` (+0.51 wins/run, -2.1 dmg/run) — strict: **not distinguishable**, substantive: **BETTER**. + +--- + +## Fresh-data confirmation (gate v2) — PRE-REGISTERED before the battles + +> **Status at pre-registration: NOT YET RUN.** This section was written and +> committed *before* any gate-v2 battle was launched. The frozen binary for +> gate v2 is built by `tournament_run.sh` from this same commit, so the +> criterion below is fixed before the data exists and cannot be moved after it. + +### Why gate v2 exists (and what it is NOT) + +Gate v1 (`## Final confirmation + SHIP`, commit `ff03e81`) required **both** +(1) the pooled 95% CI on Δwins/run excluding 0 and (2) the **plain +cross-opponent sign test** favouring `strafe` at p < 0.05. Leg 1 passed; leg 2 +failed at **10/13 decisive, p = 0.0923**. The campaign's own analyzer shows why +leg 2 was the weak link: of the three cross-opponent tests it computes, the +plain sign test is the **weakest** — it keeps only the sign of each per-opponent +delta and discards its magnitude — and at n = 13 decisive pairs it needs +**11/13** for p < 0.05. The other two tests on the *same* gate-v1 data cleared +0.05 (sign-flip permutation p = 0.0178; Wilcoxon p = 0.0189). So gate v1's +leg 2 was **over-conservative and underpowered**, not evidence that the effect +is absent. + +**The gate-v1 failure is NOT being reinterpreted.** The default is still `tfil`; +nothing in the gate-v1 section above is revised, and no gate-v1 battle is +re-used below. Gate v2 is a **new** pre-registration that (a) uses a better +primary test and (b) is confirmed on **genuinely fresh, independent data**. A +test chosen after seeing which p-value it produces would be worthless; this +section is committed first. + +### Primary test for gate v2 (pre-committed) + +`strafe` beats `tfil` on the fresh session **iff all three hold**: + +1. the **sign-flip permutation test** on the per-opponent paired Δwins/run + (`strafe` − `tfil`), two-sided, **p < 0.05**; **AND** +2. the pooled 95% CI on the mean Δwins/run **excludes 0**; **AND** +3. the point estimate is **positive** (in `strafe`'s favour). + +The sign-flip permutation test is the primary because it is the campaign's +strongest cross-opponent test that keeps the magnitude of each paired delta; it +is already implemented, deterministic-exact at n ≤ 20, and was **not** chosen by +peeking at the fresh result. (That it also cleared 0.05 on gate v1 is a +supporting fact, not the reason: the reason is that it is the power-appropriate +test for this paired design.) + +**Secondary (reported, NOT gating):** the plain cross-opponent sign test, the +Wilcoxon signed-rank test, and the damage / damage-taken / incoming-hit-rate / +mean-distance metrics. + +### The ship rule (pre-committed) + +**Ship the default flip (change `getEnv("TR_MOVEMENT", "tfil")` to `"strafe"` +in `ModularBot_garage/src/ModularBot.nim`) ONLY if the primary test passes on +the fresh data below. If it fails, do NOT ship**, record the failure, and leave +the default as `tfil`. There is no second, data-dependent choice: pass = ship, +fail = don't. + +### The fresh data (pre-committed) + +* **Genuinely fresh:** a new session (`/tmp/ab/j122_v2`), new run set, first + battle launched after this commit. No gate-v1 output is re-used or pooled. +* **Design:** 2 arms × 15 opponents × **10 runs** × 3 rounds = **300 battles** + (150 per arm) at conc 6, against the **frozen panel** + `tools/ab/panel_movement.txt`. The gate-v1 confirmation used 5 runs/arm; 10 + runs/arm halves each per-opponent delta's run noise — exactly what gate v1's + underpowered leg lacked. +* **Arms:** `strafe` (champion) and `tfil` (the arm to beat), **nothing else** — + the extra power is spent on the pair, not on a third arm. +* **Reference:** `tfil`. Every delta below is (`arm` − `tfil`). + +**Pre-registered prediction (recorded BEFORE the battles):** the sign-flip +permutation test passes at p < 0.05 with ≥ 12/15 opponents in `strafe`'s favour, +and the default is flipped to `strafe`. + diff --git a/tools/ab/arms_movement_v2.txt b/tools/ab/arms_movement_v2.txt new file mode 100644 index 0000000..cdc46c4 --- /dev/null +++ b/tools/ab/arms_movement_v2.txt @@ -0,0 +1,32 @@ +# ───────────────────────────────────────────────────────────────────────────── +# arms_movement_v2.txt — GATE-V2 fresh-data confirmation of the movement +# champion, run at the HIGHEST power of the campaign (10 runs/arm vs 5 in the +# gate-v1 confirmation) on the FROZEN panel tools/ab/panel_movement.txt. +# +# Why this file exists (see docs/movement_campaign.md, "Fresh-data confirmation +# (gate v2)", pre-registered and committed BEFORE these battles): +# * Gate v1 (commit ff03e81) required BOTH a pooled 95% CI excluding 0 AND the +# plain cross-opponent sign test at p<0.05. The CI passed; the plain sign +# test failed at 10/13 decisive, p=0.0923. The plain sign test is the +# WEAKEST of the three cross-opponent tests the analyzer prints (it discards +# the magnitude of each paired delta) and was underpowered at n=13. +# * The gate-v1 failure is NOT reinterpreted. Gate v2 is a NEW +# pre-registration with the sign-flip permutation test as the PRIMARY test, +# confirmed on FRESH, independent data. +# +# PRIMARY TEST (pre-committed): sign-flip permutation test on per-opponent +# paired Δwins/run, two-sided p<0.05 AND pooled 95% CI excluding 0 AND a +# positive point estimate. Ship the default flip ONLY if it passes. +# +# REFERENCE is `tfil` — every delta below is (arm − tfil), opponent by opponent. +# +# Format: name | ENV=value ENV=value | label +# ───────────────────────────────────────────────────────────────────────────── + +# 1. THE CHAMPION under gate-v2 confirmation — current strafe defaults. +strafe | TR_MOVEMENT=strafe | champion — current strafe defaults (candidate to ship) + +# 2. THE ARM TO BEAT — the currently shipped default, explicitly selected. +# (The explicit `TR_MOVEMENT=tfil` also proves the override path stays +# reachable after any default flip.) +tfil | TR_MOVEMENT=tfil | shipped baseline — explicit tfil override \ No newline at end of file