movement gate v2: pre-register the sign-flip primary test on fresh 300-battle data (before any battle)
This commit is contained in:
@@ -1474,3 +1474,79 @@ PRIMARY metrics are dmg/run and wins/run; hit rate is never the verdict. The pre
|
||||
Reference `tfil`: 115.3 dmg/run, 1.21 wins/run, 17.40% incoming, 384 px.
|
||||
|
||||
Highest wins delta: `wide_spread` (+0.51 wins/run, -2.1 dmg/run) — strict: **not distinguishable**, substantive: **BETTER**.
|
||||
|
||||
---
|
||||
|
||||
## Fresh-data confirmation (gate v2) — PRE-REGISTERED before the battles
|
||||
|
||||
> **Status at pre-registration: NOT YET RUN.** This section was written and
|
||||
> committed *before* any gate-v2 battle was launched. The frozen binary for
|
||||
> gate v2 is built by `tournament_run.sh` from this same commit, so the
|
||||
> criterion below is fixed before the data exists and cannot be moved after it.
|
||||
|
||||
### Why gate v2 exists (and what it is NOT)
|
||||
|
||||
Gate v1 (`## Final confirmation + SHIP`, commit `ff03e81`) required **both**
|
||||
(1) the pooled 95% CI on Δwins/run excluding 0 and (2) the **plain
|
||||
cross-opponent sign test** favouring `strafe` at p < 0.05. Leg 1 passed; leg 2
|
||||
failed at **10/13 decisive, p = 0.0923**. The campaign's own analyzer shows why
|
||||
leg 2 was the weak link: of the three cross-opponent tests it computes, the
|
||||
plain sign test is the **weakest** — it keeps only the sign of each per-opponent
|
||||
delta and discards its magnitude — and at n = 13 decisive pairs it needs
|
||||
**11/13** for p < 0.05. The other two tests on the *same* gate-v1 data cleared
|
||||
0.05 (sign-flip permutation p = 0.0178; Wilcoxon p = 0.0189). So gate v1's
|
||||
leg 2 was **over-conservative and underpowered**, not evidence that the effect
|
||||
is absent.
|
||||
|
||||
**The gate-v1 failure is NOT being reinterpreted.** The default is still `tfil`;
|
||||
nothing in the gate-v1 section above is revised, and no gate-v1 battle is
|
||||
re-used below. Gate v2 is a **new** pre-registration that (a) uses a better
|
||||
primary test and (b) is confirmed on **genuinely fresh, independent data**. A
|
||||
test chosen after seeing which p-value it produces would be worthless; this
|
||||
section is committed first.
|
||||
|
||||
### Primary test for gate v2 (pre-committed)
|
||||
|
||||
`strafe` beats `tfil` on the fresh session **iff all three hold**:
|
||||
|
||||
1. the **sign-flip permutation test** on the per-opponent paired Δwins/run
|
||||
(`strafe` − `tfil`), two-sided, **p < 0.05**; **AND**
|
||||
2. the pooled 95% CI on the mean Δwins/run **excludes 0**; **AND**
|
||||
3. the point estimate is **positive** (in `strafe`'s favour).
|
||||
|
||||
The sign-flip permutation test is the primary because it is the campaign's
|
||||
strongest cross-opponent test that keeps the magnitude of each paired delta; it
|
||||
is already implemented, deterministic-exact at n ≤ 20, and was **not** chosen by
|
||||
peeking at the fresh result. (That it also cleared 0.05 on gate v1 is a
|
||||
supporting fact, not the reason: the reason is that it is the power-appropriate
|
||||
test for this paired design.)
|
||||
|
||||
**Secondary (reported, NOT gating):** the plain cross-opponent sign test, the
|
||||
Wilcoxon signed-rank test, and the damage / damage-taken / incoming-hit-rate /
|
||||
mean-distance metrics.
|
||||
|
||||
### The ship rule (pre-committed)
|
||||
|
||||
**Ship the default flip (change `getEnv("TR_MOVEMENT", "tfil")` to `"strafe"`
|
||||
in `ModularBot_garage/src/ModularBot.nim`) ONLY if the primary test passes on
|
||||
the fresh data below. If it fails, do NOT ship**, record the failure, and leave
|
||||
the default as `tfil`. There is no second, data-dependent choice: pass = ship,
|
||||
fail = don't.
|
||||
|
||||
### The fresh data (pre-committed)
|
||||
|
||||
* **Genuinely fresh:** a new session (`/tmp/ab/j122_v2`), new run set, first
|
||||
battle launched after this commit. No gate-v1 output is re-used or pooled.
|
||||
* **Design:** 2 arms × 15 opponents × **10 runs** × 3 rounds = **300 battles**
|
||||
(150 per arm) at conc 6, against the **frozen panel**
|
||||
`tools/ab/panel_movement.txt`. The gate-v1 confirmation used 5 runs/arm; 10
|
||||
runs/arm halves each per-opponent delta's run noise — exactly what gate v1's
|
||||
underpowered leg lacked.
|
||||
* **Arms:** `strafe` (champion) and `tfil` (the arm to beat), **nothing else** —
|
||||
the extra power is spent on the pair, not on a third arm.
|
||||
* **Reference:** `tfil`. Every delta below is (`arm` − `tfil`).
|
||||
|
||||
**Pre-registered prediction (recorded BEFORE the battles):** the sign-flip
|
||||
permutation test passes at p < 0.05 with ≥ 12/15 opponents in `strafe`'s favour,
|
||||
and the default is flipped to `strafe`.
|
||||
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# arms_movement_v2.txt — GATE-V2 fresh-data confirmation of the movement
|
||||
# champion, run at the HIGHEST power of the campaign (10 runs/arm vs 5 in the
|
||||
# gate-v1 confirmation) on the FROZEN panel tools/ab/panel_movement.txt.
|
||||
#
|
||||
# Why this file exists (see docs/movement_campaign.md, "Fresh-data confirmation
|
||||
# (gate v2)", pre-registered and committed BEFORE these battles):
|
||||
# * Gate v1 (commit ff03e81) required BOTH a pooled 95% CI excluding 0 AND the
|
||||
# plain cross-opponent sign test at p<0.05. The CI passed; the plain sign
|
||||
# test failed at 10/13 decisive, p=0.0923. The plain sign test is the
|
||||
# WEAKEST of the three cross-opponent tests the analyzer prints (it discards
|
||||
# the magnitude of each paired delta) and was underpowered at n=13.
|
||||
# * The gate-v1 failure is NOT reinterpreted. Gate v2 is a NEW
|
||||
# pre-registration with the sign-flip permutation test as the PRIMARY test,
|
||||
# confirmed on FRESH, independent data.
|
||||
#
|
||||
# PRIMARY TEST (pre-committed): sign-flip permutation test on per-opponent
|
||||
# paired Δwins/run, two-sided p<0.05 AND pooled 95% CI excluding 0 AND a
|
||||
# positive point estimate. Ship the default flip ONLY if it passes.
|
||||
#
|
||||
# REFERENCE is `tfil` — every delta below is (arm − tfil), opponent by opponent.
|
||||
#
|
||||
# Format: name | ENV=value ENV=value | label
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
# 1. THE CHAMPION under gate-v2 confirmation — current strafe defaults.
|
||||
strafe | TR_MOVEMENT=strafe | champion — current strafe defaults (candidate to ship)
|
||||
|
||||
# 2. THE ARM TO BEAT — the currently shipped default, explicitly selected.
|
||||
# (The explicit `TR_MOVEMENT=tfil` also proves the override path stays
|
||||
# reachable after any default flip.)
|
||||
tfil | TR_MOVEMENT=tfil | shipped baseline — explicit tfil override
|
||||
Reference in New Issue
Block a user