movement gate v2: pre-register the sign-flip primary test on fresh 300-battle data (before any battle)

This commit is contained in:
2026-09-26 02:44:22 +02:00
parent feefc1912e
commit 514674886d
2 changed files with 108 additions and 0 deletions
+76
View File
@@ -1474,3 +1474,79 @@ PRIMARY metrics are dmg/run and wins/run; hit rate is never the verdict. The pre
Reference `tfil`: 115.3 dmg/run, 1.21 wins/run, 17.40% incoming, 384 px.
Highest wins delta: `wide_spread` (+0.51 wins/run, -2.1 dmg/run) — strict: **not distinguishable**, substantive: **BETTER**.
---
## Fresh-data confirmation (gate v2) — PRE-REGISTERED before the battles
> **Status at pre-registration: NOT YET RUN.** This section was written and
> committed *before* any gate-v2 battle was launched. The frozen binary for
> gate v2 is built by `tournament_run.sh` from this same commit, so the
> criterion below is fixed before the data exists and cannot be moved after it.
### Why gate v2 exists (and what it is NOT)
Gate v1 (`## Final confirmation + SHIP`, commit `ff03e81`) required **both**
(1) the pooled 95% CI on Δwins/run excluding 0 and (2) the **plain
cross-opponent sign test** favouring `strafe` at p < 0.05. Leg 1 passed; leg 2
failed at **10/13 decisive, p = 0.0923**. The campaign's own analyzer shows why
leg 2 was the weak link: of the three cross-opponent tests it computes, the
plain sign test is the **weakest** — it keeps only the sign of each per-opponent
delta and discards its magnitude — and at n = 13 decisive pairs it needs
**11/13** for p < 0.05. The other two tests on the *same* gate-v1 data cleared
0.05 (sign-flip permutation p = 0.0178; Wilcoxon p = 0.0189). So gate v1's
leg 2 was **over-conservative and underpowered**, not evidence that the effect
is absent.
**The gate-v1 failure is NOT being reinterpreted.** The default is still `tfil`;
nothing in the gate-v1 section above is revised, and no gate-v1 battle is
re-used below. Gate v2 is a **new** pre-registration that (a) uses a better
primary test and (b) is confirmed on **genuinely fresh, independent data**. A
test chosen after seeing which p-value it produces would be worthless; this
section is committed first.
### Primary test for gate v2 (pre-committed)
`strafe` beats `tfil` on the fresh session **iff all three hold**:
1. the **sign-flip permutation test** on the per-opponent paired Δwins/run
(`strafe` − `tfil`), two-sided, **p < 0.05**; **AND**
2. the pooled 95% CI on the mean Δwins/run **excludes 0**; **AND**
3. the point estimate is **positive** (in `strafe`'s favour).
The sign-flip permutation test is the primary because it is the campaign's
strongest cross-opponent test that keeps the magnitude of each paired delta; it
is already implemented, deterministic-exact at n ≤ 20, and was **not** chosen by
peeking at the fresh result. (That it also cleared 0.05 on gate v1 is a
supporting fact, not the reason: the reason is that it is the power-appropriate
test for this paired design.)
**Secondary (reported, NOT gating):** the plain cross-opponent sign test, the
Wilcoxon signed-rank test, and the damage / damage-taken / incoming-hit-rate /
mean-distance metrics.
### The ship rule (pre-committed)
**Ship the default flip (change `getEnv("TR_MOVEMENT", "tfil")` to `"strafe"`
in `ModularBot_garage/src/ModularBot.nim`) ONLY if the primary test passes on
the fresh data below. If it fails, do NOT ship**, record the failure, and leave
the default as `tfil`. There is no second, data-dependent choice: pass = ship,
fail = don't.
### The fresh data (pre-committed)
* **Genuinely fresh:** a new session (`/tmp/ab/j122_v2`), new run set, first
battle launched after this commit. No gate-v1 output is re-used or pooled.
* **Design:** 2 arms × 15 opponents × **10 runs** × 3 rounds = **300 battles**
(150 per arm) at conc 6, against the **frozen panel**
`tools/ab/panel_movement.txt`. The gate-v1 confirmation used 5 runs/arm; 10
runs/arm halves each per-opponent delta's run noise — exactly what gate v1's
underpowered leg lacked.
* **Arms:** `strafe` (champion) and `tfil` (the arm to beat), **nothing else** —
the extra power is spent on the pair, not on a third arm.
* **Reference:** `tfil`. Every delta below is (`arm` − `tfil`).
**Pre-registered prediction (recorded BEFORE the battles):** the sign-flip
permutation test passes at p < 0.05 with ≥ 12/15 opponents in `strafe`'s favour,
and the default is flipped to `strafe`.