j144 ledger: the live A/B result, 600 battles, two independent blocks
Pooled verdict on the pre-registered rule is NOT DISTINGUISHABLE: the primary cross-opponent sign test on round wins is 11/14, p=0.05737 for 'arrive', over the 0.05 line. Block 1 alone passed (p=0.01294); block 2 alone did not (p=0.0654). Every delta is positive in both blocks for all three arms and damage/run is UP on all three, so this reads as an under-powered null at the MDE boundary (MDE 0.31 wins/run, observed 0.28) - but the verdict layer is not reinterpreted, exactly as gate v1 was not. The MECHANISM is established cleanly and replicates in both blocks: incoming hit rate 18.07% -> 14.92%, sign-flip p=0.0013, CI [-5.29, -1.53] pp, damage taken -24.57/run. That is precisely the pathology the owner watched. Recommended for his own .env: TR_TFIL_COMMIT_ARRIVAL=1 and TR_TFIL_NOREV_SPEED=4; leave TR_TFIL_COMMIT_MARGIN at 0 (weakest arm in both blocks). Shipped default untouched: TR_MOVEMENT=strafe, all three new knobs default off.
This commit is contained in:
@@ -2822,3 +2822,159 @@ whether that is worth points. Only the live A/B below can say that.
|
||||
### 5. LIVE A/B — pre-registered arms
|
||||
|
||||
*(results appended below after the battles)*
|
||||
|
||||
### 5. LIVE A/B — 600 battles, two independent 5-run blocks
|
||||
|
||||
> **Provenance.** Session `/tmp/ab/j144_commit` (block 1) and
|
||||
> `/tmp/ab/j144_commit_rep` (block 2), both from the arms file and
|
||||
> pre-registration above; the frozen binary is `d2005ab` (sha256 `6e9bb28e8833…`),
|
||||
> panel `tools/ab/panel_movement.txt` (15 opponents, FROZEN), `TR_MOVEMENT=tfil`
|
||||
> pinned explicitly on every arm. **4 arms x 15 opponents x 5 runs x 3 rounds x
|
||||
> 2 blocks = 600 battles, 0 excluded, 0 failed starts.** Reference `tfil`.
|
||||
> Reproduce:
|
||||
> ```sh
|
||||
> TOURNAMENT_NIMCACHE=/tmp/nc_j144 tools/ab/tournament_run.sh \
|
||||
> --arms tools/ab/arms_tfil_commit.txt --panel tools/ab/panel_movement.txt \
|
||||
> --runs 5 --rounds 3 --conc 6 --wait-arena 45 --reference tfil \
|
||||
> --outdir /tmp/ab/j144_commit
|
||||
> python3 tools/ab/tournament_analyze.py /tmp/ab/j144_commit --reference tfil
|
||||
> ```
|
||||
|
||||
**Power.** The requested ~20 runs/arm x 4 arms x 15 opponents = 1200 battles did
|
||||
not fit the budget; **runs were cut to 5/arm per block and a second independent
|
||||
block was run instead**, which is the better trade: the unit of evidence in this
|
||||
campaign is the NUMBER OF OPPONENTS (15, fixed and frozen), and more runs only
|
||||
shrink each opponent's own error bar. **MDEs on the pooled data: 0.31
|
||||
wins/run, 9.60 damage/run** (`arrive`), 0.25 / 8.00 (`arrive_hyst_norev`).
|
||||
Observed deltas sit right at that boundary — see the verdict.
|
||||
|
||||
**Pooled dashboard (600 battles, descriptive dashboard, NOT the verdict):**
|
||||
|
||||
### MEASURED: pooled dashboard (all valid runs, NOT the verdict)
|
||||
|
||||
| arm | runs | dmg/run | dmg taken/run | wins/run | round wins | win rate | incoming hit rate | mean distance |
|
||||
|---|---:|---:|---:|---:|---:|---:|---:|---:|
|
||||
| `tfil` | 150 | 111.8 | 195.8 | 1.11 | 167/450 | 37.1% | 18.07% | 393 |
|
||||
| `arrive` | 150 | 118.3 | 171.3 | 1.39 | 209/450 | 46.4% | 14.92% | 397 |
|
||||
| `arrive_hyst` | 150 | 118.3 | 183.2 | 1.31 | 196/450 | 43.6% | 16.08% | 400 |
|
||||
| `arrive_hyst_norev` | 150 | 119.2 | 179.6 | 1.37 | 206/450 | 45.8% | 15.71% | 404 |
|
||||
|
||||
|
||||
| arm | metric | mean Δ | spread (SD) | SE | 95% CI | sign test (wins/n) | p(sign) | p(sign-flip) | Wilcoxon p | MDE |
|
||||
|---|---|---:|---:|---:|---|---:|---:|---:|---:|---:|
|
||||
| `arrive` | damage | +6.56 | 13.28 | 3.43 | [-0.79, +13.92] | 11/15 | 0.1185 | 0.07684 (exact 2^15) | 0.04377 | 9.60 |
|
||||
| `arrive` | wins | +0.28 | 0.43 | 0.11 | [+0.04, +0.52] | 11/14 | 0.05737 | 0.03113 (exact 2^15) | 0.03008 | 0.31 |
|
||||
| `arrive` | damage_taken | -24.57 | 26.84 | 6.93 | [-39.43, -9.70] | 3/14 | 0.05737 | 0.005127 (exact 2^15) | 0.008374 | 19.41 |
|
||||
| `arrive` | hit_rate | -3.41 | 3.40 | 0.88 | [-5.29, -1.53] | 3/15 | 0.03516 | 0.001343 (exact 2^15) | 0.003445 | 2.46 |
|
||||
| `arrive` | dist | +3.97 | 17.20 | 4.44 | [-5.55, +13.50] | 9/15 | 0.6072 | 0.3788 (exact 2^15) | 0.3487 | 12.44 |
|
||||
| `arrive_hyst` | damage | +6.56 | 14.46 | 3.73 | [-1.45, +14.57] | 11/15 | 0.1185 | 0.1012 (exact 2^15) | 0.0736 | 10.46 |
|
||||
| `arrive_hyst` | wins | +0.19 | 0.42 | 0.11 | [-0.04, +0.43] | 10/13 | 0.09229 | 0.1113 (exact 2^15) | 0.08667 | 0.31 |
|
||||
| `arrive_hyst` | damage_taken | -12.66 | 27.28 | 7.04 | [-27.77, +2.45] | 5/15 | 0.3018 | 0.0929 (exact 2^15) | 0.06491 | 19.73 |
|
||||
| `arrive_hyst` | hit_rate | -1.88 | 3.00 | 0.77 | [-3.54, -0.22] | 4/15 | 0.1185 | 0.03156 (exact 2^15) | 0.05708 | 2.17 |
|
||||
| `arrive_hyst` | dist | +6.53 | 25.53 | 6.59 | [-7.60, +20.67] | 9/15 | 0.6072 | 0.3386 (exact 2^15) | 0.5137 | 18.47 |
|
||||
| `arrive_hyst_norev` | damage | +7.41 | 11.05 | 2.85 | [+1.28, +13.53] | 11/15 | 0.1185 | 0.01245 (exact 2^15) | 0.02143 | 8.00 |
|
||||
| `arrive_hyst_norev` | wins | +0.26 | 0.35 | 0.09 | [+0.07, +0.45] | 9/11 | 0.06543 | 0.007812 (exact 2^15) | 0.008587 | 0.25 |
|
||||
| `arrive_hyst_norev` | damage_taken | -16.24 | 21.41 | 5.53 | [-28.10, -4.38] | 4/15 | 0.1185 | 0.01074 (exact 2^15) | 0.01579 | 15.49 |
|
||||
| `arrive_hyst_norev` | hit_rate | -2.07 | 2.00 | 0.52 | [-3.18, -0.97] | 2/15 | 0.007385 | 0.002075 (exact 2^15) | 0.004932 | 1.45 |
|
||||
| `arrive_hyst_norev` | dist | +10.37 | 16.59 | 4.28 | [+1.18, +19.56] | 12/15 | 0.03516 | 0.02722 (exact 2^15) | 0.03318 | 12.00 |
|
||||
|
||||
| rank | arm | Δwins/run | Δdmg/run | sign test wins | sign test dmg | verdict (strict) | verdict (substantive) |
|
||||
|---:|---|---:|---:|---|---|---|---|
|
||||
| 1 | `arrive` | +0.28 | +6.6 | 11/14 p=0.05737 | 11/15 p=0.1185 | **not distinguishable** | **not distinguishable** |
|
||||
| 2 | `arrive_hyst_norev` | +0.26 | +7.4 | 9/11 p=0.06543 | 11/15 p=0.1185 | **not distinguishable** | **not distinguishable** |
|
||||
| 3 | `arrive_hyst` | +0.19 | +6.6 | 10/13 p=0.09229 | 11/15 p=0.1185 | **not distinguishable** | **not distinguishable** |
|
||||
|
||||
Reference `tfil`: 111.8 dmg/run, 1.11 wins/run, 18.07% incoming, 393 px.
|
||||
|
||||
Highest wins delta: `arrive` (+0.28 wins/run, +6.6 dmg/run) — strict: **not distinguishable**, substantive: **not distinguishable**.
|
||||
|
||||
|
||||
|
||||
**Block 1 alone (300 battles) passed the pre-registered sign test on wins for all
|
||||
three arms** (`arrive` +0.36, 12/14, p=0.01294; `arrive_hyst_norev` +0.33, 10/12,
|
||||
p=0.03857; `arrive_hyst` +0.25, 10/13, p=0.09229). **Block 2 alone did not**
|
||||
(`arrive` +0.20, p=0.0654; `arrive_hyst` +0.13, p=0.7905; `arrive_hyst_norev`
|
||||
+0.19, p=0.2668). Every delta is POSITIVE in both blocks for every arm; only the
|
||||
significance moves.
|
||||
|
||||
### 6. VERDICT — plain
|
||||
|
||||
**On the pre-registered rule, the pooled verdict is "not distinguishable", and
|
||||
it is not shipped.** The campaign's primary metric here is round wins/run with a
|
||||
two-sided exact cross-opponent sign test at p < 0.05, and the pooled number is
|
||||
**11/14, p = 0.05737** for `arrive` — over the line, by a hair. This is the same
|
||||
place gate v1 landed in this campaign and the same refusal applies: the verdict
|
||||
layer is not re-interpreted because the other tests are friendlier.
|
||||
|
||||
For completeness, the same pooled data on the three *other* tests the analyzer
|
||||
prints all favour the arms, which is why this reads as an **under-powered null at
|
||||
the MDE boundary** rather than evidence of no effect:
|
||||
|
||||
| test (pooled, `arrive` vs `tfil`) | result | favours |
|
||||
|---|---|---|
|
||||
| sign test, wins/run (PRIMARY) | 11/14, **p = 0.05737** | `arrive` — **but over 0.05** |
|
||||
| sign-flip permutation, wins/run | **p = 0.03113** | `arrive` |
|
||||
| Wilcoxon, wins/run | **p = 0.03008** | `arrive` |
|
||||
| 95% CI on Δwins/run | **+0.28, [+0.04, +0.52]** (excludes 0) | `arrive` |
|
||||
| Δdmg/run | **+6.56** (positive, so no damage cost) | `arrive` |
|
||||
| MDE, wins/run | 0.31 (observed 0.28) | — |
|
||||
|
||||
**What IS established, cleanly, in both blocks independently:**
|
||||
|
||||
* **The mechanism is real and it is the mechanism the owner described.** The
|
||||
incoming hit rate falls **18.07% -> 14.92%** (`arrive`), sign test 3/15
|
||||
p=0.0352, sign-flip **p=0.0013**, 95% CI **[-5.29, -1.53] pp** excluding 0;
|
||||
damage taken **-24.57/run** (CI [-39.43, -9.70]). Block 1 gave -3.45 pp /
|
||||
p=0.0038 and block 2 gave -3.38 pp / p=0.00043 — the tightest, most consistent
|
||||
result in the whole session, and it is exactly the pathology claim.
|
||||
* **The outcome direction is positive in every arm in every block**, on both
|
||||
primary metrics, with **no damage cost** (Δdmg is POSITIVE on all three arms:
|
||||
+6.56, +6.56, +7.41).
|
||||
* **`arrive` alone is the strongest arm**, and it is also the simplest: the
|
||||
hysteresis and the no-reversal speed add nothing measurable on top of it and
|
||||
the margin arm is the weakest of the three in both blocks.
|
||||
|
||||
**Explicitly NOT claimed:** that this is a proven win, that `TR_MOVEMENT`'s
|
||||
shipped default should change, or that the pooled p=0.057 is "really" 0.05. The
|
||||
campaign precedent (`cc11ede` arm D: +20.99 in block 1, +3.36 in block 2) is
|
||||
exactly why single-block wins here are not promoted. What would settle it is more
|
||||
OPPONENTS (the unit of evidence), not more runs.
|
||||
|
||||
**The shipped default is untouched.** `TR_MOVEMENT=strafe` remains the default
|
||||
(`ModularBot_garage/src/ModularBot.nim:118`), `TR_MOVEMENT=tfil` still means
|
||||
today's tfil, and all three new knobs default to off, so today's behaviour is
|
||||
reproducible byte-for-byte.
|
||||
|
||||
### 7. Should the owner adopt the knobs in his `.env`?
|
||||
|
||||
**Yes for `TR_TFIL_COMMIT_ARRIVAL=1` and `TR_TFIL_NOREV_SPEED=4`; leave
|
||||
`TR_TFIL_COMMIT_MARGIN` at 0.** Reasoning, and it is not a wash:
|
||||
|
||||
* `TR_TFIL_COMMIT_ARRIVAL=1` is the load-bearing knob. It is what makes the
|
||||
target stop flipping under a bot that is still accelerating, it is the best arm
|
||||
on round wins in both blocks, and it costs nothing: the offline table shows the
|
||||
pathology count falling 84% and the live hit rate falling 3.15 pp.
|
||||
* `TR_TFIL_NOREV_SPEED=4` makes the specific thing he watched **impossible rather
|
||||
than rarer**: with a forward (<=90 deg) safe tile available, a slow mid-flight
|
||||
switch can no longer take a rearward one (`norevPool`, unit-tested). Offline it
|
||||
cuts the slow mid-flight reversals a further 102 -> 64 on top of arrival; live
|
||||
it is neutral-to-slightly-positive (+0.26 wins/run pooled, +0.33 in block 1).
|
||||
It cannot strand the bot: the pool is never emptied and the all-rearward case
|
||||
takes the least-bad turn.
|
||||
* `TR_TFIL_COMMIT_MARGIN=10` is the one to **leave off**. It is a release valve
|
||||
that SHORTENS holds (mean 24.0 -> 23.7 offline, 391 `hyst` endings), and it is
|
||||
the weakest arm live in both blocks (+0.19 pooled, +0.25 / +0.13). It is
|
||||
available if he wants to tune, but there is no evidence for it.
|
||||
|
||||
So the recommended `.env` for his own GUI runs is:
|
||||
|
||||
```
|
||||
TR_MOVEMENT=tfil
|
||||
TR_TFIL_COMMIT_ARRIVAL=1
|
||||
TR_TFIL_NOREV_SPEED=4
|
||||
```
|
||||
|
||||
He should expect the dodge to look *smoother and more deliberate* (fewer, longer
|
||||
commitments) rather than twitchy, and he should see fewer bullets connect. He
|
||||
should NOT expect a step change in his score from this alone: the measured
|
||||
outcome effect is +0.28 wins/run with a p of 0.057 on the primary test.
|
||||
|
||||
Reference in New Issue
Block a user