movement gate v2: pre-register the sign-flip primary test on fresh 300-battle data (before any battle)
This commit is contained in:
@@ -1474,3 +1474,79 @@ PRIMARY metrics are dmg/run and wins/run; hit rate is never the verdict. The pre
|
|||||||
Reference `tfil`: 115.3 dmg/run, 1.21 wins/run, 17.40% incoming, 384 px.
|
Reference `tfil`: 115.3 dmg/run, 1.21 wins/run, 17.40% incoming, 384 px.
|
||||||
|
|
||||||
Highest wins delta: `wide_spread` (+0.51 wins/run, -2.1 dmg/run) — strict: **not distinguishable**, substantive: **BETTER**.
|
Highest wins delta: `wide_spread` (+0.51 wins/run, -2.1 dmg/run) — strict: **not distinguishable**, substantive: **BETTER**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Fresh-data confirmation (gate v2) — PRE-REGISTERED before the battles
|
||||||
|
|
||||||
|
> **Status at pre-registration: NOT YET RUN.** This section was written and
|
||||||
|
> committed *before* any gate-v2 battle was launched. The frozen binary for
|
||||||
|
> gate v2 is built by `tournament_run.sh` from this same commit, so the
|
||||||
|
> criterion below is fixed before the data exists and cannot be moved after it.
|
||||||
|
|
||||||
|
### Why gate v2 exists (and what it is NOT)
|
||||||
|
|
||||||
|
Gate v1 (`## Final confirmation + SHIP`, commit `ff03e81`) required **both**
|
||||||
|
(1) the pooled 95% CI on Δwins/run excluding 0 and (2) the **plain
|
||||||
|
cross-opponent sign test** favouring `strafe` at p < 0.05. Leg 1 passed; leg 2
|
||||||
|
failed at **10/13 decisive, p = 0.0923**. The campaign's own analyzer shows why
|
||||||
|
leg 2 was the weak link: of the three cross-opponent tests it computes, the
|
||||||
|
plain sign test is the **weakest** — it keeps only the sign of each per-opponent
|
||||||
|
delta and discards its magnitude — and at n = 13 decisive pairs it needs
|
||||||
|
**11/13** for p < 0.05. The other two tests on the *same* gate-v1 data cleared
|
||||||
|
0.05 (sign-flip permutation p = 0.0178; Wilcoxon p = 0.0189). So gate v1's
|
||||||
|
leg 2 was **over-conservative and underpowered**, not evidence that the effect
|
||||||
|
is absent.
|
||||||
|
|
||||||
|
**The gate-v1 failure is NOT being reinterpreted.** The default is still `tfil`;
|
||||||
|
nothing in the gate-v1 section above is revised, and no gate-v1 battle is
|
||||||
|
re-used below. Gate v2 is a **new** pre-registration that (a) uses a better
|
||||||
|
primary test and (b) is confirmed on **genuinely fresh, independent data**. A
|
||||||
|
test chosen after seeing which p-value it produces would be worthless; this
|
||||||
|
section is committed first.
|
||||||
|
|
||||||
|
### Primary test for gate v2 (pre-committed)
|
||||||
|
|
||||||
|
`strafe` beats `tfil` on the fresh session **iff all three hold**:
|
||||||
|
|
||||||
|
1. the **sign-flip permutation test** on the per-opponent paired Δwins/run
|
||||||
|
(`strafe` − `tfil`), two-sided, **p < 0.05**; **AND**
|
||||||
|
2. the pooled 95% CI on the mean Δwins/run **excludes 0**; **AND**
|
||||||
|
3. the point estimate is **positive** (in `strafe`'s favour).
|
||||||
|
|
||||||
|
The sign-flip permutation test is the primary because it is the campaign's
|
||||||
|
strongest cross-opponent test that keeps the magnitude of each paired delta; it
|
||||||
|
is already implemented, deterministic-exact at n ≤ 20, and was **not** chosen by
|
||||||
|
peeking at the fresh result. (That it also cleared 0.05 on gate v1 is a
|
||||||
|
supporting fact, not the reason: the reason is that it is the power-appropriate
|
||||||
|
test for this paired design.)
|
||||||
|
|
||||||
|
**Secondary (reported, NOT gating):** the plain cross-opponent sign test, the
|
||||||
|
Wilcoxon signed-rank test, and the damage / damage-taken / incoming-hit-rate /
|
||||||
|
mean-distance metrics.
|
||||||
|
|
||||||
|
### The ship rule (pre-committed)
|
||||||
|
|
||||||
|
**Ship the default flip (change `getEnv("TR_MOVEMENT", "tfil")` to `"strafe"`
|
||||||
|
in `ModularBot_garage/src/ModularBot.nim`) ONLY if the primary test passes on
|
||||||
|
the fresh data below. If it fails, do NOT ship**, record the failure, and leave
|
||||||
|
the default as `tfil`. There is no second, data-dependent choice: pass = ship,
|
||||||
|
fail = don't.
|
||||||
|
|
||||||
|
### The fresh data (pre-committed)
|
||||||
|
|
||||||
|
* **Genuinely fresh:** a new session (`/tmp/ab/j122_v2`), new run set, first
|
||||||
|
battle launched after this commit. No gate-v1 output is re-used or pooled.
|
||||||
|
* **Design:** 2 arms × 15 opponents × **10 runs** × 3 rounds = **300 battles**
|
||||||
|
(150 per arm) at conc 6, against the **frozen panel**
|
||||||
|
`tools/ab/panel_movement.txt`. The gate-v1 confirmation used 5 runs/arm; 10
|
||||||
|
runs/arm halves each per-opponent delta's run noise — exactly what gate v1's
|
||||||
|
underpowered leg lacked.
|
||||||
|
* **Arms:** `strafe` (champion) and `tfil` (the arm to beat), **nothing else** —
|
||||||
|
the extra power is spent on the pair, not on a third arm.
|
||||||
|
* **Reference:** `tfil`. Every delta below is (`arm` − `tfil`).
|
||||||
|
|
||||||
|
**Pre-registered prediction (recorded BEFORE the battles):** the sign-flip
|
||||||
|
permutation test passes at p < 0.05 with ≥ 12/15 opponents in `strafe`'s favour,
|
||||||
|
and the default is flipped to `strafe`.
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,32 @@
|
|||||||
|
# ─────────────────────────────────────────────────────────────────────────────
|
||||||
|
# arms_movement_v2.txt — GATE-V2 fresh-data confirmation of the movement
|
||||||
|
# champion, run at the HIGHEST power of the campaign (10 runs/arm vs 5 in the
|
||||||
|
# gate-v1 confirmation) on the FROZEN panel tools/ab/panel_movement.txt.
|
||||||
|
#
|
||||||
|
# Why this file exists (see docs/movement_campaign.md, "Fresh-data confirmation
|
||||||
|
# (gate v2)", pre-registered and committed BEFORE these battles):
|
||||||
|
# * Gate v1 (commit ff03e81) required BOTH a pooled 95% CI excluding 0 AND the
|
||||||
|
# plain cross-opponent sign test at p<0.05. The CI passed; the plain sign
|
||||||
|
# test failed at 10/13 decisive, p=0.0923. The plain sign test is the
|
||||||
|
# WEAKEST of the three cross-opponent tests the analyzer prints (it discards
|
||||||
|
# the magnitude of each paired delta) and was underpowered at n=13.
|
||||||
|
# * The gate-v1 failure is NOT reinterpreted. Gate v2 is a NEW
|
||||||
|
# pre-registration with the sign-flip permutation test as the PRIMARY test,
|
||||||
|
# confirmed on FRESH, independent data.
|
||||||
|
#
|
||||||
|
# PRIMARY TEST (pre-committed): sign-flip permutation test on per-opponent
|
||||||
|
# paired Δwins/run, two-sided p<0.05 AND pooled 95% CI excluding 0 AND a
|
||||||
|
# positive point estimate. Ship the default flip ONLY if it passes.
|
||||||
|
#
|
||||||
|
# REFERENCE is `tfil` — every delta below is (arm − tfil), opponent by opponent.
|
||||||
|
#
|
||||||
|
# Format: name | ENV=value ENV=value | label
|
||||||
|
# ─────────────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
# 1. THE CHAMPION under gate-v2 confirmation — current strafe defaults.
|
||||||
|
strafe | TR_MOVEMENT=strafe | champion — current strafe defaults (candidate to ship)
|
||||||
|
|
||||||
|
# 2. THE ARM TO BEAT — the currently shipped default, explicitly selected.
|
||||||
|
# (The explicit `TR_MOVEMENT=tfil` also proves the override path stays
|
||||||
|
# reachable after any default flip.)
|
||||||
|
tfil | TR_MOVEMENT=tfil | shipped baseline — explicit tfil override
|
||||||
Reference in New Issue
Block a user