movement final: confirmation result and ship decision (gate failed on the sign-test leg; default NOT flipped)
This commit is contained in:
+300
-22
@@ -13,6 +13,173 @@ gun."*
|
||||
|
||||
---
|
||||
|
||||
## OUTCOME — FINAL (read this first)
|
||||
|
||||
**The best measured 1v1 movement is `TR_MOVEMENT=strafe`; the pre-registered
|
||||
SHIP GATE was NOT met, so the shipped default is UNCHANGED (`tfil`).**
|
||||
|
||||
The higher-power confirmation (225 battles, 5 runs/arm, 0 invalid) measured
|
||||
`strafe` over `tfil` at **Δwins/run +0.33, 95% CI [+0.08, +0.58]** (the CI
|
||||
excludes 0 — leg 1 passes), with the incoming hit rate down **−4.75 pp** and
|
||||
**−37.7 damage taken/run** at no detectable damage cost — but the **exact
|
||||
two-sided cross-opponent sign test is 10/13, p = 0.0923**, which does **not**
|
||||
clear the pre-registered p < 0.05 (leg 2 fails). Both legs were required, so
|
||||
**no default flip was made and no binary was rebuilt**. Refusing to ship when
|
||||
the pre-registered criterion fails is the task's own rule and is a **successful
|
||||
outcome**. The champion is confirmed as the best movement; it stays fully
|
||||
reachable as an explicit opt-in: **`TR_MOVEMENT=strafe`**.
|
||||
|
||||
The full criterion, the confirmation tables (per-opponent, sign test, CI, MDE)
|
||||
and the ship decision follow directly below; the raw analyzer output is at the
|
||||
bottom of this file.
|
||||
|
||||
---
|
||||
|
||||
## Final confirmation + SHIP
|
||||
|
||||
> **Provenance.** The ship criterion below was pre-registered and committed in
|
||||
> `ff03e81` *before* the confirmation battles ran; that commit is the frozen
|
||||
> binary's source (`ff03e81591fc…`, binary sha256 `4757a734f3b0…`). Session
|
||||
> `/tmp/ab/j120_final`, arms file `tools/ab/arms_movement_final.txt`, frozen
|
||||
> panel `tools/ab/panel_movement.txt`, **3 arms × 15 opponents × 5 runs × 3
|
||||
> rounds = 225 battles**, conc 6, `--reference tfil`; **0 invalid runs, 0 failed
|
||||
> starts**. Power is higher than every previous batch (5 runs/arm vs 3).
|
||||
|
||||
**THE PRE-REGISTERED SHIP CRITERION (fixed before fighting).** Ship the flip of
|
||||
`TR_MOVEMENT`'s default from `tfil` to `strafe` **only if**, on this frozen
|
||||
panel and one frozen binary, `strafe` beats `tfil` head-to-head on round wins/run
|
||||
with **both**: (1) the 95% CI on Δwins/run **excluding 0**; **and** (2) the
|
||||
cross-opponent **sign test favouring `strafe` at p < 0.05** (the campaign's
|
||||
standing convention, two-sided exact binomial).
|
||||
|
||||
**SHIP DECISION: NO — the gate failed on the sign-test leg.** (MEASURED)
|
||||
|
||||
| leg | test | result | verdict |
|
||||
|---|---|---|---|
|
||||
| 1 | 95% CI on Δwins/run (`strafe` − `tfil`) | **+0.33, 95% CI [+0.08, +0.58]** — excludes 0 | **PASS** |
|
||||
| 2 | cross-opponent sign test (exact, two-sided) | **10/13 decisive opponents, p = 0.0923** | **FAIL** (p > 0.05) |
|
||||
|
||||
Both legs were required, so **the default was NOT flipped and the shipped binary
|
||||
was NOT rebuilt** — `TR_MOVEMENT` still defaults to `tfil`. Per Task 1's own
|
||||
rule, refusing to ship when the criterion fails is a successful outcome.
|
||||
|
||||
### The confirmation numbers (MEASURED, 225 battles, 0 excluded)
|
||||
|
||||
Pooled dashboard (descriptive, NOT the verdict):
|
||||
|
||||
| arm | runs | dmg/run | dmg taken/run | wins/run | round wins | win rate | incoming hit rate | mean distance |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| `strafe` (champion) | 75 | 107.5 | 153.0 | 1.55 | 116/225 | 51.6% | 13.10% | 429 |
|
||||
| `tfil` (shipped) | 75 | 115.3 | 190.7 | 1.21 | 91/225 | 40.4% | 17.40% | 384 |
|
||||
| `wide_spread` (challenger) | 75 | 113.2 | 147.6 | 1.72 | 129/225 | 57.3% | 12.53% | 426 |
|
||||
|
||||
Per-opponent Δwins/run (arm − `tfil`), the unit of evidence:
|
||||
|
||||
| opponent | style | `strafe` | `wide_spread` |
|
||||
|---|---|---:|---:|
|
||||
| DrussGT | dodger | -0.20 | +0.00 |
|
||||
| Diamond | dodger | +0.00 | +0.00 |
|
||||
| Dookious | dodger | +0.40 | +0.80 |
|
||||
| GresSuffurd | dodger | +1.20 | +0.80 |
|
||||
| CassiusClay | dodger | +0.80 | +0.40 |
|
||||
| RetroGirl | pattern | +0.20 | +0.20 |
|
||||
| TripHammer | pattern | +0.80 | +1.20 |
|
||||
| Coriantumr | pattern | +1.00 | +1.40 |
|
||||
| WallAvoider | wallfollower | -0.20 | -0.20 |
|
||||
| HawkOnFire | cornercamper | +0.20 | +1.20 |
|
||||
| SpinBot | spinner | +0.00 | +0.00 |
|
||||
| DiamondStealer | rammer | -0.20 | -0.60 |
|
||||
| BlitzBat | brawler | +0.60 | +0.20 |
|
||||
| YersiniaPestis | aggressive | +0.20 | +1.00 |
|
||||
| Ascendant | aggressive | +0.20 | +1.20 |
|
||||
|
||||
Cross-opponent aggregation (the verdict layer; `spread` = SD across opponents):
|
||||
|
||||
| arm | metric | mean Δ | spread | SE | 95% CI | sign test (wins/n) | p(sign) | p(sign-flip) | Wilcoxon p | MDE |
|
||||
|---|---|---:|---:|---:|---|---:|---:|---:|---:|---:|
|
||||
| `strafe` | wins | +0.33 | 0.45 | 0.12 | [+0.08, +0.58] | 10/13 | **0.0923** | 0.0178 | 0.0189 | 0.33 |
|
||||
| `strafe` | damage | -7.85 | 16.33 | 4.22 | [-16.89, +1.20] | 5/15 | 0.3018 | 0.0843 | 0.0832 | 11.81 |
|
||||
| `strafe` | damage_taken | -37.74 | 32.07 | 8.28 | [-55.50, -19.97] | 1/15 | 0.00098 | 0.00037 | 0.0016 | 23.20 |
|
||||
| `strafe` | hit_rate | -4.75 | 3.41 | 0.88 | [-6.64, -2.86] | 1/15 | 0.00098 | 0.00018 | 0.0011 | 2.47 |
|
||||
| `strafe` | dist | +44.78 | 32.42 | 8.37 | [+26.82, +62.73] | 15/15 | 6e-5 | 6e-5 | 7e-4 | 23.45 |
|
||||
| `wide_spread` | wins | +0.51 | 0.62 | 0.16 | [+0.16, +0.85] | 10/12 | 0.0386 | 0.0103 | 0.0120 | 0.45 |
|
||||
| `wide_spread` | damage | -2.08 | 15.15 | 3.91 | [-10.47, +6.31] | 6/15 | 0.6072 | 0.6038 | 0.5895 | 10.96 |
|
||||
|
||||
### Reading (MEASURED / INFERRED)
|
||||
|
||||
* **MEASURED — the champion is confirmed on effect size and mechanism.**
|
||||
`strafe` wins **+0.33 wins/run** over `tfil` (CI [+0.08, +0.58]); the
|
||||
incoming hit rate is down **−4.75 pp** (CI [−6.64, −2.86]; 1/15 opponents
|
||||
favour `tfil`) and **−37.7 damage taken/run** (CI [−55.50, −19.97]) at a
|
||||
damage cost of −7.85/run, **inside the MDE (11.81) so not detectable**. This
|
||||
is the **fifth independent session** to show the same survival win (previous
|
||||
four: 53.0% vs 42.2%, 52.6% vs 38.5%, and Batches 1–2's +0.33…+0.58).
|
||||
* **MEASURED — the gate is a sign-test near-miss, not a contradiction.** Ten of
|
||||
thirteen decisive opponents favour `strafe`; two ties (Diamond, SpinBot — both
|
||||
≈0 wins/run for `tfil`) and three **exactly −0.20** opponents (DrussGT,
|
||||
WallAvoider, DiamondStealer — each −1 round out of 15) leave the exact
|
||||
binomial at p = 0.0923. The **sign-flip permutation (p = 0.0178)** and
|
||||
**Wilcoxon (p = 0.0189)** — the campaign's other two cross-opponent tests —
|
||||
both clear 0.05; only the exact sign test does not. I did **not** move the
|
||||
pre-registered goalpost: the gate as committed required the sign test, so the
|
||||
ship did not happen.
|
||||
* **MEASURED — the challenger nominally out-scored the champion in this
|
||||
session.** `wide_spread` posted the session's best wins/run (+0.51, CI
|
||||
[+0.16, +0.85], sign test 10/12 p = 0.0386) and was damage-neutral (−2.08,
|
||||
inside its MDE). In Batches 3–4 it was a tie (+0.11, p = 0.55). **INFERRED:**
|
||||
the two arms are **not separable** head-to-head in this design (both sit
|
||||
~+0.3…+0.5 over `tfil`); the confirmation does not install `wide_spread` as a
|
||||
better champion — it merely fails to separate it.
|
||||
* **MEASURED — `tfil` is the worst of the three on both primaries**: 40.4% round
|
||||
wins vs 51.6% (`strafe`) and 57.3% (`wide_spread`), and the highest incoming
|
||||
hit rate (17.40%). The direction of the whole campaign is unchanged.
|
||||
|
||||
### What the default is now, and how to use/revert it
|
||||
|
||||
**The default is `TR_MOVEMENT=tfil` — UNCHANGED.** No source line was edited and
|
||||
no binary was rebuilt, so the shipped `ModularBot_garage/out/ModularBot` is the
|
||||
same binary as before this job. (The single dispatch line that would flip the
|
||||
default is `ModularBot_garage/src/ModularBot.nim:118`:
|
||||
`let MovementName* = getEnv("TR_MOVEMENT", "tfil")…`.)
|
||||
|
||||
To run the confirmed-best movement, opt in with an env-only switch (both engines
|
||||
are always compiled in, so no rebuild is needed):
|
||||
|
||||
```sh
|
||||
TR_MOVEMENT=strafe
|
||||
```
|
||||
|
||||
`tfil` remains the default and is a one-word revert (`TR_MOVEMENT=tfil`); an
|
||||
unrecognised value still falls back to `tfil`.
|
||||
|
||||
### Honest limits (what this design space did NOT cover)
|
||||
|
||||
* **The failed gate is a discrete sign test on 15 opponents.** At n = 13
|
||||
decisive, 10/13 is one opponent short of the 11/13 needed for two-sided
|
||||
p < 0.05; the effect (51.6% vs 40.4% round-win rate) and its CI are
|
||||
unambiguous. Settling the sign test would need a **pre-registered larger
|
||||
panel** — adding opponents now would start a new panel and re-open every prior
|
||||
verdict, so it was not done.
|
||||
* **The confirmation could not resolve `strafe` vs `wide_spread`** (+0.18
|
||||
wins/run apart, overlapping CIs).
|
||||
* **Scope:** 1v1, 800×600, these 15 opponents. Nothing here is evidence about
|
||||
melee (a different game — see j116), the twins/smaller arena, or opponents
|
||||
harder than this panel.
|
||||
* **Local optimum:** the strafe retune is the measured optimum only *of the knobs
|
||||
that were swept* (reversal dwell, spread/reach, heat strength, wall geometry).
|
||||
A structurally different mover (wave surfer, learned policy) is untested.
|
||||
* **Round wins here are survival wins.** The measured advantage is "takes fewer,
|
||||
weaker hits and survives more rounds at unchanged damage output", not "kills
|
||||
faster"; it need not transfer to an opponent that wins on damage.
|
||||
|
||||
**Pre-registered prediction — WRONG, recorded as wrong.** I predicted `strafe`
|
||||
would pass both legs and ship, and that `wide_spread` would tie it. `strafe`
|
||||
passed leg 1 but **failed leg 2** (p = 0.0923), so the ship prediction is wrong;
|
||||
`wide_spread` was nominally *above* `strafe` (+0.51 vs +0.33), wrong in
|
||||
direction though correct that no separation exists.
|
||||
|
||||
---
|
||||
|
||||
## 0. The one caveat this campaign exists to close
|
||||
|
||||
Everything measured about movement before this campaign is **DrussGT-only**:
|
||||
@@ -1164,35 +1331,146 @@ the default flip is a separate, explicit decision and was NOT made here.
|
||||
|---|---|---:|---|---|
|
||||
| `/tmp/ab/j119_b3` | `1256357` | 270 (0 failed; 1 excluded: Ascendant/strafe r1) | strafe, tfil, fast_flip, slow_flip, wide_spread, narrow | nothing beats strafe; wide_spread +0.11 NS (p=0.55) |
|
||||
| `/tmp/ab/j119_b4` | `1256357` | 270 (0 failed, 0 excluded) | strafe, tfil, bullet_strong, field_strong, field_off, wall_tight | nothing beats strafe; field_strong and field_off detectably WORSE |
|
||||
| `/tmp/ab/j120_final` | `ff03e81` | 225 (0 failed, 0 excluded) | strafe, tfil, wide_spread | **ship gate FAILED on the sign-test leg** (10/13, p=0.0923); default NOT flipped |
|
||||
|
||||
---
|
||||
|
||||
## Final confirmation + SHIP — PRE-REGISTRATION
|
||||
## Raw analyzer report (verbatim) — session `/tmp/ab/j120_final`
|
||||
|
||||
> **PRE-REGISTERED BEFORE THE BATTLES (this section is committed, and its commit
|
||||
> is the frozen binary's source, before the first battle is launched).** Session
|
||||
> `/tmp/ab/j120_final`. Arms file `tools/ab/arms_movement_final.txt`, panel
|
||||
> `tools/ab/panel_movement.txt` (frozen), **3 arms × 15 opponents × 5 runs × 3
|
||||
> rounds = 225 battles**, conc 6, `--reference tfil`.
|
||||
*(The ship criterion was pre-registered and committed at `ff03e81` before these
|
||||
battles ran. The curated decision is the `## Final confirmation + SHIP` section
|
||||
at the top of this file; this is the analyzer's unedited output.)
|
||||
|
||||
**THE PRE-REGISTERED SHIP CRITERION (fixed before fighting).** Ship the flip of
|
||||
`TR_MOVEMENT`'s default from `tfil` to `strafe` **only if**, on this frozen
|
||||
panel and this one frozen binary, `strafe` beats `tfil` head-to-head on the
|
||||
PRIMARY metric *round wins/run* with **both**:
|
||||
|
||||
1. the **95% CI on Δwins/run excluding 0**; **and**
|
||||
2. the **cross-opponent sign test favouring `strafe`** (p < 0.05).
|
||||
### MEASURED: session
|
||||
|
||||
This is stricter than the campaign's §1 rule 2 (which would let a survival win
|
||||
carry a damage-neutral arm): the ship decision requires the *win* metric itself
|
||||
to clear both bars. If it does not pass, **do not ship** — say so and stop; that
|
||||
is a fully successful outcome. This batch uses **5 runs/arm (vs 3 in every
|
||||
previous batch)** for higher power; `wide_spread` (the only positive point
|
||||
estimate from Batches 3–4, +0.11/run, NS) rides along as the single challenger.
|
||||
* commit `ff03e81591fc28efa16cf5f7bb00a4d0f5d47590`, frozen binary sha256 `4757a734f3b0…`
|
||||
* 15 opponents × 3 arms × 5 runs × 3 rounds = 225 battles, conc=6
|
||||
* arms file `arms_movement_final.txt`, panel file `panel_movement.txt`
|
||||
* reference arm: **`tfil`** — every delta below is (arm − tfil), opponent by opponent
|
||||
|
||||
**Pre-registered prediction (before the battles):** `strafe` passes both bars
|
||||
and ships; `wide_spread` ties `strafe` (no separable win); `tfil` is not
|
||||
preferred by any primary metric. A wrong prediction is recorded as wrong.
|
||||
* liveness: 0 run(s) excluded (225 total)
|
||||
|
||||
**Outcome:** pending — filled in below after the session.
|
||||
### MEASURED: per-opponent paired table (per arm)
|
||||
|
||||
#### `strafe` — champion — current strafe defaults (candidate to ship) (paired on 15 opponents)
|
||||
|
||||
| opponent | style | dmg/run ref→arm | Δdmg | wins/run ref→arm | Δwins | Δdmg taken | Δhit rate (pp) | dist ref→arm |
|
||||
|---|---|---:|---:|---:|---:|---:|---:|---:|
|
||||
| DrussGT | dodger | 126.9→104.9 | -22.1 | 1.20→1.00 | -0.20 | -23.7 | -1.65 | 441→504 |
|
||||
| Diamond | dodger | 54.0→65.0 | +11.0 | 0.20→0.20 | +0.00 | -52.8 | -5.09 | 446→483 |
|
||||
| Dookious | dodger | 99.7→95.4 | -4.3 | 1.40→1.80 | +0.40 | -10.2 | -2.28 | 435→442 |
|
||||
| GresSuffurd | dodger | 109.3→127.4 | +18.2 | 1.20→2.40 | +1.20 | -67.7 | -5.74 | 420→426 |
|
||||
| CassiusClay | dodger | 68.9→80.5 | +11.6 | 0.40→1.20 | +0.80 | -36.3 | -5.59 | 370→395 |
|
||||
| RetroGirl | pattern | 167.4→155.2 | -12.2 | 1.80→2.00 | +0.20 | -32.7 | -4.16 | 384→443 |
|
||||
| TripHammer | pattern | 66.5→49.3 | -17.2 | 0.00→0.80 | +0.80 | -49.3 | -4.22 | 460→492 |
|
||||
| Coriantumr | pattern | 68.3→77.6 | +9.3 | 0.60→1.60 | +1.00 | -56.6 | -5.54 | 412→464 |
|
||||
| WallAvoider | wallfollower | 178.8→153.9 | -24.9 | 2.40→2.20 | -0.20 | -10.0 | -3.56 | 301→320 |
|
||||
| HawkOnFire | cornercamper | 119.6→116.7 | -2.9 | 1.60→1.80 | +0.20 | -37.6 | -5.10 | 419→481 |
|
||||
| SpinBot | spinner | 290.6→265.9 | -24.7 | 3.00→3.00 | +0.00 | +22.4 | +2.03 | 280→411 |
|
||||
| DiamondStealer | rammer | 176.4→143.4 | -33.0 | 1.60→1.40 | -0.20 | -18.4 | -2.16 | 236→263 |
|
||||
| BlitzBat | brawler | 71.6→45.3 | -26.3 | 2.20→2.80 | +0.60 | -120.8 | -13.39 | 448→529 |
|
||||
| YersiniaPestis | aggressive | 63.4→60.2 | -3.2 | 0.40→0.60 | +0.20 | -44.5 | -8.17 | 364→411 |
|
||||
| Ascendant | aggressive | 68.1→71.0 | +2.9 | 0.20→0.40 | +0.20 | -27.7 | -6.68 | 350→373 |
|
||||
|
||||
#### `tfil` — shipped baseline — explicit tfil override (paired on 15 opponents)
|
||||
|
||||
| opponent | style | dmg/run ref→arm | Δdmg | wins/run ref→arm | Δwins | Δdmg taken | Δhit rate (pp) | dist ref→arm |
|
||||
|---|---|---:|---:|---:|---:|---:|---:|---:|
|
||||
| DrussGT | dodger | 126.9→126.9 | +0.0 | 1.20→1.20 | +0.00 | +0.0 | +0.00 | 441→441 |
|
||||
| Diamond | dodger | 54.0→54.0 | +0.0 | 0.20→0.20 | +0.00 | +0.0 | +0.00 | 446→446 |
|
||||
| Dookious | dodger | 99.7→99.7 | +0.0 | 1.40→1.40 | +0.00 | +0.0 | +0.00 | 435→435 |
|
||||
| GresSuffurd | dodger | 109.3→109.3 | +0.0 | 1.20→1.20 | +0.00 | +0.0 | +0.00 | 420→420 |
|
||||
| CassiusClay | dodger | 68.9→68.9 | +0.0 | 0.40→0.40 | +0.00 | +0.0 | +0.00 | 370→370 |
|
||||
| RetroGirl | pattern | 167.4→167.4 | +0.0 | 1.80→1.80 | +0.00 | +0.0 | +0.00 | 384→384 |
|
||||
| TripHammer | pattern | 66.5→66.5 | +0.0 | 0.00→0.00 | +0.00 | +0.0 | +0.00 | 460→460 |
|
||||
| Coriantumr | pattern | 68.3→68.3 | +0.0 | 0.60→0.60 | +0.00 | +0.0 | +0.00 | 412→412 |
|
||||
| WallAvoider | wallfollower | 178.8→178.8 | +0.0 | 2.40→2.40 | +0.00 | +0.0 | +0.00 | 301→301 |
|
||||
| HawkOnFire | cornercamper | 119.6→119.6 | +0.0 | 1.60→1.60 | +0.00 | +0.0 | +0.00 | 419→419 |
|
||||
| SpinBot | spinner | 290.6→290.6 | +0.0 | 3.00→3.00 | +0.00 | +0.0 | +0.00 | 280→280 |
|
||||
| DiamondStealer | rammer | 176.4→176.4 | +0.0 | 1.60→1.60 | +0.00 | +0.0 | +0.00 | 236→236 |
|
||||
| BlitzBat | brawler | 71.6→71.6 | +0.0 | 2.20→2.20 | +0.00 | +0.0 | +0.00 | 448→448 |
|
||||
| YersiniaPestis | aggressive | 63.4→63.4 | +0.0 | 0.40→0.40 | +0.00 | +0.0 | +0.00 | 364→364 |
|
||||
| Ascendant | aggressive | 68.1→68.1 | +0.0 | 0.20→0.20 | +0.00 | +0.0 | +0.00 | 350→350 |
|
||||
|
||||
#### `wide_spread` — Batches 3–4 positive-point challenger (±2 tiles, 216px) (paired on 15 opponents)
|
||||
|
||||
| opponent | style | dmg/run ref→arm | Δdmg | wins/run ref→arm | Δwins | Δdmg taken | Δhit rate (pp) | dist ref→arm |
|
||||
|---|---|---:|---:|---:|---:|---:|---:|---:|
|
||||
| DrussGT | dodger | 126.9→113.9 | -13.0 | 1.20→1.20 | +0.00 | -31.0 | -1.95 | 441→479 |
|
||||
| Diamond | dodger | 54.0→85.0 | +31.0 | 0.20→0.20 | +0.00 | -83.5 | -7.08 | 446→505 |
|
||||
| Dookious | dodger | 99.7→101.9 | +2.2 | 1.40→2.20 | +0.80 | -28.4 | -3.76 | 435→460 |
|
||||
| GresSuffurd | dodger | 109.3→114.7 | +5.4 | 1.20→2.00 | +0.80 | -39.2 | -4.59 | 420→446 |
|
||||
| CassiusClay | dodger | 68.9→66.8 | -2.1 | 0.40→0.80 | +0.40 | -35.7 | -4.69 | 370→377 |
|
||||
| RetroGirl | pattern | 167.4→151.1 | -16.2 | 1.80→2.00 | +0.20 | -20.2 | -3.39 | 384→432 |
|
||||
| TripHammer | pattern | 66.5→69.7 | +3.2 | 0.00→1.20 | +1.20 | -69.6 | -5.55 | 460→486 |
|
||||
| Coriantumr | pattern | 68.3→82.1 | +13.8 | 0.60→2.00 | +1.40 | -58.7 | -6.23 | 412→478 |
|
||||
| WallAvoider | wallfollower | 178.8→173.8 | -5.0 | 2.40→2.20 | -0.20 | -5.7 | -0.40 | 301→310 |
|
||||
| HawkOnFire | cornercamper | 119.6→113.7 | -5.9 | 1.60→2.80 | +1.20 | -81.2 | -7.64 | 419→468 |
|
||||
| SpinBot | spinner | 290.6→286.4 | -4.2 | 3.00→3.00 | +0.00 | +22.4 | +4.08 | 280→378 |
|
||||
| DiamondStealer | rammer | 176.4→150.8 | -25.5 | 1.60→1.00 | -0.60 | +13.1 | -0.42 | 236→256 |
|
||||
| BlitzBat | brawler | 71.6→44.3 | -27.3 | 2.20→2.40 | +0.20 | -93.0 | -11.49 | 448→529 |
|
||||
| YersiniaPestis | aggressive | 63.4→62.9 | -0.5 | 0.40→1.40 | +1.00 | -68.6 | -9.48 | 364→413 |
|
||||
| Ascendant | aggressive | 68.1→81.1 | +13.1 | 0.20→1.40 | +1.20 | -68.1 | -10.98 | 350→376 |
|
||||
|
||||
### MEASURED: pooled dashboard (all valid runs, NOT the verdict)
|
||||
|
||||
| arm | runs | dmg/run | dmg taken/run | wins/run | round wins | win rate | incoming hit rate | mean distance |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| `strafe` | 75 | 107.5 | 153.0 | 1.55 | 116/225 | 51.6% | 13.10% | 429 |
|
||||
| `tfil` | 75 | 115.3 | 190.7 | 1.21 | 91/225 | 40.4% | 17.40% | 384 |
|
||||
| `wide_spread` | 75 | 113.2 | 147.6 | 1.72 | 129/225 | 57.3% | 12.53% | 426 |
|
||||
|
||||
### MEASURED: cross-opponent aggregation (the verdict layer)
|
||||
|
||||
Deltas are per-opponent (arm − reference). `spread` is the SD of those deltas ACROSS opponents; `SE` = spread/√n; `95% CI` = mean ± t·SE. Sign test = how many opponents the arm wins (ties dropped), exact binomial; sign-flip = permutation test on the mean of the deltas.
|
||||
|
||||
| arm | metric | mean Δ | spread (SD) | SE | 95% CI | sign test (wins/n) | p(sign) | p(sign-flip) | Wilcoxon p | MDE |
|
||||
|---|---|---:|---:|---:|---|---:|---:|---:|---:|---:|
|
||||
| `strafe` | damage | -7.85 | 16.33 | 4.22 | [-16.89, +1.20] | 5/15 | 0.3018 | 0.08429 (exact 2^15) | 0.08322 | 11.81 |
|
||||
| `strafe` | wins | +0.33 | 0.45 | 0.12 | [+0.08, +0.58] | 10/13 | 0.09229 | 0.01782 (exact 2^15) | 0.01886 | 0.33 |
|
||||
| `strafe` | damage_taken | -37.74 | 32.07 | 8.28 | [-55.50, -19.97] | 1/15 | 0.0009766 | 0.0003662 (exact 2^15) | 0.001621 | 23.20 |
|
||||
| `strafe` | hit_rate | -4.75 | 3.41 | 0.88 | [-6.64, -2.86] | 1/15 | 0.0009766 | 0.0001831 (exact 2^15) | 0.001092 | 2.47 |
|
||||
| `strafe` | dist | +44.78 | 32.42 | 8.37 | [+26.82, +62.73] | 15/15 | 6.104e-05 | 6.104e-05 (exact 2^15) | 0.0007265 | 23.45 |
|
||||
| `wide_spread` | damage | -2.08 | 15.15 | 3.91 | [-10.47, +6.31] | 6/15 | 0.6072 | 0.6038 (exact 2^15) | 0.5895 | 10.96 |
|
||||
| `wide_spread` | wins | +0.51 | 0.62 | 0.16 | [+0.16, +0.85] | 10/12 | 0.03857 | 0.01025 (exact 2^15) | 0.012 | 0.45 |
|
||||
| `wide_spread` | damage_taken | -43.15 | 35.44 | 9.15 | [-62.78, -23.52] | 2/15 | 0.007385 | 0.0007935 (exact 2^15) | 0.002377 | 25.64 |
|
||||
| `wide_spread` | hit_rate | -4.91 | 4.22 | 1.09 | [-7.24, -2.57] | 1/15 | 0.0009766 | 0.0007935 (exact 2^15) | 0.002377 | 3.05 |
|
||||
| `wide_spread` | dist | +41.82 | 26.12 | 6.74 | [+27.35, +56.28] | 15/15 | 6.104e-05 | 6.104e-05 (exact 2^15) | 0.0007265 | 18.89 |
|
||||
|
||||
#### By inferred style (explanation only, never the verdict)
|
||||
|
||||
| arm | style | n | mean Δdmg | mean Δwins | mean Δhit rate (pp) |
|
||||
|---|---|---:|---:|---:|---:|
|
||||
| `strafe` | aggressive | 2 | -0.1 | +0.20 | -7.43 |
|
||||
| `strafe` | brawler | 1 | -26.3 | +0.60 | -13.39 |
|
||||
| `strafe` | cornercamper | 1 | -2.9 | +0.20 | -5.10 |
|
||||
| `strafe` | dodger | 5 | +2.9 | +0.44 | -4.07 |
|
||||
| `strafe` | pattern | 3 | -6.7 | +0.67 | -4.64 |
|
||||
| `strafe` | rammer | 1 | -33.0 | -0.20 | -2.16 |
|
||||
| `strafe` | spinner | 1 | -24.7 | +0.00 | +2.03 |
|
||||
| `strafe` | wallfollower | 1 | -24.9 | -0.20 | -3.56 |
|
||||
| `wide_spread` | aggressive | 2 | +6.3 | +1.10 | -10.23 |
|
||||
| `wide_spread` | brawler | 1 | -27.3 | +0.20 | -11.49 |
|
||||
| `wide_spread` | cornercamper | 1 | -5.9 | +1.20 | -7.64 |
|
||||
| `wide_spread` | dodger | 5 | +4.7 | +0.40 | -4.41 |
|
||||
| `wide_spread` | pattern | 3 | +0.3 | +0.93 | -5.06 |
|
||||
| `wide_spread` | rammer | 1 | -25.5 | -0.60 | -0.42 |
|
||||
| `wide_spread` | spinner | 1 | -4.2 | +0.00 | +4.08 |
|
||||
| `wide_spread` | wallfollower | 1 | -5.0 | -0.20 | -0.40 |
|
||||
|
||||
### The pre-registered verdict (rules fixed in `docs/movement_campaign.md`)
|
||||
|
||||
PRIMARY metrics are dmg/run and wins/run; hit rate is never the verdict. The pre-registered rule says an arm is BETTER when one primary metric is UP at sign-test p<0.05 `while the other does not go down`. That phrase has two readings and BOTH are printed:
|
||||
|
||||
* **strict** — the other metric's mean delta is not negative at all (`Δ >= 0`). Nothing can be BETTER while it costs *any* mean damage.
|
||||
* **substantive** — the other metric's delta is not *detectably* down: the sign test is not significant **and** the delta is smaller than that metric's MDE (the pre-registered rule 3 says an effect under the MDE is not detectable, so it cannot count as a loss).
|
||||
|
||||
| rank | arm | Δwins/run | Δdmg/run | sign test wins | sign test dmg | verdict (strict) | verdict (substantive) |
|
||||
|---:|---|---:|---:|---|---|---|---|
|
||||
| 1 | `wide_spread` | +0.51 | -2.1 | 10/12 p=0.03857 | 6/15 p=0.6072 | **not distinguishable** | **BETTER** |
|
||||
| 2 | `strafe` | +0.33 | -7.8 | 10/13 p=0.09229 | 5/15 p=0.3018 | **not distinguishable** | **not distinguishable** |
|
||||
|
||||
Reference `tfil`: 115.3 dmg/run, 1.21 wins/run, 17.40% incoming, 384 px.
|
||||
|
||||
Highest wins delta: `wide_spread` (+0.51 wins/run, -2.1 dmg/run) — strict: **not distinguishable**, substantive: **BETTER**.
|
||||
|
||||
Reference in New Issue
Block a user