From 0e7e124c7fbdcda3fd9406b7c82f1ace3936c32f Mon Sep 17 00:00:00 2001 From: Davide Cappellini Date: Sat, 26 Sep 2026 21:43:03 +0200 Subject: [PATCH] j144 ledger: the live A/B result, 600 battles, two independent blocks Pooled verdict on the pre-registered rule is NOT DISTINGUISHABLE: the primary cross-opponent sign test on round wins is 11/14, p=0.05737 for 'arrive', over the 0.05 line. Block 1 alone passed (p=0.01294); block 2 alone did not (p=0.0654). Every delta is positive in both blocks for all three arms and damage/run is UP on all three, so this reads as an under-powered null at the MDE boundary (MDE 0.31 wins/run, observed 0.28) - but the verdict layer is not reinterpreted, exactly as gate v1 was not. The MECHANISM is established cleanly and replicates in both blocks: incoming hit rate 18.07% -> 14.92%, sign-flip p=0.0013, CI [-5.29, -1.53] pp, damage taken -24.57/run. That is precisely the pathology the owner watched. Recommended for his own .env: TR_TFIL_COMMIT_ARRIVAL=1 and TR_TFIL_NOREV_SPEED=4; leave TR_TFIL_COMMIT_MARGIN at 0 (weakest arm in both blocks). Shipped default untouched: TR_MOVEMENT=strafe, all three new knobs default off. --- docs/movement_campaign.md | 156 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 156 insertions(+) diff --git a/docs/movement_campaign.md b/docs/movement_campaign.md index 5e5b683..e654d0a 100644 --- a/docs/movement_campaign.md +++ b/docs/movement_campaign.md @@ -2822,3 +2822,159 @@ whether that is worth points. Only the live A/B below can say that. ### 5. LIVE A/B — pre-registered arms *(results appended below after the battles)* + +### 5. LIVE A/B — 600 battles, two independent 5-run blocks + +> **Provenance.** Session `/tmp/ab/j144_commit` (block 1) and +> `/tmp/ab/j144_commit_rep` (block 2), both from the arms file and +> pre-registration above; the frozen binary is `d2005ab` (sha256 `6e9bb28e8833…`), +> panel `tools/ab/panel_movement.txt` (15 opponents, FROZEN), `TR_MOVEMENT=tfil` +> pinned explicitly on every arm. **4 arms x 15 opponents x 5 runs x 3 rounds x +> 2 blocks = 600 battles, 0 excluded, 0 failed starts.** Reference `tfil`. +> Reproduce: +> ```sh +> TOURNAMENT_NIMCACHE=/tmp/nc_j144 tools/ab/tournament_run.sh \ +> --arms tools/ab/arms_tfil_commit.txt --panel tools/ab/panel_movement.txt \ +> --runs 5 --rounds 3 --conc 6 --wait-arena 45 --reference tfil \ +> --outdir /tmp/ab/j144_commit +> python3 tools/ab/tournament_analyze.py /tmp/ab/j144_commit --reference tfil +> ``` + +**Power.** The requested ~20 runs/arm x 4 arms x 15 opponents = 1200 battles did +not fit the budget; **runs were cut to 5/arm per block and a second independent +block was run instead**, which is the better trade: the unit of evidence in this +campaign is the NUMBER OF OPPONENTS (15, fixed and frozen), and more runs only +shrink each opponent's own error bar. **MDEs on the pooled data: 0.31 +wins/run, 9.60 damage/run** (`arrive`), 0.25 / 8.00 (`arrive_hyst_norev`). +Observed deltas sit right at that boundary — see the verdict. + +**Pooled dashboard (600 battles, descriptive dashboard, NOT the verdict):** + +### MEASURED: pooled dashboard (all valid runs, NOT the verdict) + +| arm | runs | dmg/run | dmg taken/run | wins/run | round wins | win rate | incoming hit rate | mean distance | +|---|---:|---:|---:|---:|---:|---:|---:|---:| +| `tfil` | 150 | 111.8 | 195.8 | 1.11 | 167/450 | 37.1% | 18.07% | 393 | +| `arrive` | 150 | 118.3 | 171.3 | 1.39 | 209/450 | 46.4% | 14.92% | 397 | +| `arrive_hyst` | 150 | 118.3 | 183.2 | 1.31 | 196/450 | 43.6% | 16.08% | 400 | +| `arrive_hyst_norev` | 150 | 119.2 | 179.6 | 1.37 | 206/450 | 45.8% | 15.71% | 404 | + + +| arm | metric | mean Δ | spread (SD) | SE | 95% CI | sign test (wins/n) | p(sign) | p(sign-flip) | Wilcoxon p | MDE | +|---|---|---:|---:|---:|---|---:|---:|---:|---:|---:| +| `arrive` | damage | +6.56 | 13.28 | 3.43 | [-0.79, +13.92] | 11/15 | 0.1185 | 0.07684 (exact 2^15) | 0.04377 | 9.60 | +| `arrive` | wins | +0.28 | 0.43 | 0.11 | [+0.04, +0.52] | 11/14 | 0.05737 | 0.03113 (exact 2^15) | 0.03008 | 0.31 | +| `arrive` | damage_taken | -24.57 | 26.84 | 6.93 | [-39.43, -9.70] | 3/14 | 0.05737 | 0.005127 (exact 2^15) | 0.008374 | 19.41 | +| `arrive` | hit_rate | -3.41 | 3.40 | 0.88 | [-5.29, -1.53] | 3/15 | 0.03516 | 0.001343 (exact 2^15) | 0.003445 | 2.46 | +| `arrive` | dist | +3.97 | 17.20 | 4.44 | [-5.55, +13.50] | 9/15 | 0.6072 | 0.3788 (exact 2^15) | 0.3487 | 12.44 | +| `arrive_hyst` | damage | +6.56 | 14.46 | 3.73 | [-1.45, +14.57] | 11/15 | 0.1185 | 0.1012 (exact 2^15) | 0.0736 | 10.46 | +| `arrive_hyst` | wins | +0.19 | 0.42 | 0.11 | [-0.04, +0.43] | 10/13 | 0.09229 | 0.1113 (exact 2^15) | 0.08667 | 0.31 | +| `arrive_hyst` | damage_taken | -12.66 | 27.28 | 7.04 | [-27.77, +2.45] | 5/15 | 0.3018 | 0.0929 (exact 2^15) | 0.06491 | 19.73 | +| `arrive_hyst` | hit_rate | -1.88 | 3.00 | 0.77 | [-3.54, -0.22] | 4/15 | 0.1185 | 0.03156 (exact 2^15) | 0.05708 | 2.17 | +| `arrive_hyst` | dist | +6.53 | 25.53 | 6.59 | [-7.60, +20.67] | 9/15 | 0.6072 | 0.3386 (exact 2^15) | 0.5137 | 18.47 | +| `arrive_hyst_norev` | damage | +7.41 | 11.05 | 2.85 | [+1.28, +13.53] | 11/15 | 0.1185 | 0.01245 (exact 2^15) | 0.02143 | 8.00 | +| `arrive_hyst_norev` | wins | +0.26 | 0.35 | 0.09 | [+0.07, +0.45] | 9/11 | 0.06543 | 0.007812 (exact 2^15) | 0.008587 | 0.25 | +| `arrive_hyst_norev` | damage_taken | -16.24 | 21.41 | 5.53 | [-28.10, -4.38] | 4/15 | 0.1185 | 0.01074 (exact 2^15) | 0.01579 | 15.49 | +| `arrive_hyst_norev` | hit_rate | -2.07 | 2.00 | 0.52 | [-3.18, -0.97] | 2/15 | 0.007385 | 0.002075 (exact 2^15) | 0.004932 | 1.45 | +| `arrive_hyst_norev` | dist | +10.37 | 16.59 | 4.28 | [+1.18, +19.56] | 12/15 | 0.03516 | 0.02722 (exact 2^15) | 0.03318 | 12.00 | + +| rank | arm | Δwins/run | Δdmg/run | sign test wins | sign test dmg | verdict (strict) | verdict (substantive) | +|---:|---|---:|---:|---|---|---|---| +| 1 | `arrive` | +0.28 | +6.6 | 11/14 p=0.05737 | 11/15 p=0.1185 | **not distinguishable** | **not distinguishable** | +| 2 | `arrive_hyst_norev` | +0.26 | +7.4 | 9/11 p=0.06543 | 11/15 p=0.1185 | **not distinguishable** | **not distinguishable** | +| 3 | `arrive_hyst` | +0.19 | +6.6 | 10/13 p=0.09229 | 11/15 p=0.1185 | **not distinguishable** | **not distinguishable** | + +Reference `tfil`: 111.8 dmg/run, 1.11 wins/run, 18.07% incoming, 393 px. + +Highest wins delta: `arrive` (+0.28 wins/run, +6.6 dmg/run) — strict: **not distinguishable**, substantive: **not distinguishable**. + + + +**Block 1 alone (300 battles) passed the pre-registered sign test on wins for all +three arms** (`arrive` +0.36, 12/14, p=0.01294; `arrive_hyst_norev` +0.33, 10/12, +p=0.03857; `arrive_hyst` +0.25, 10/13, p=0.09229). **Block 2 alone did not** +(`arrive` +0.20, p=0.0654; `arrive_hyst` +0.13, p=0.7905; `arrive_hyst_norev` ++0.19, p=0.2668). Every delta is POSITIVE in both blocks for every arm; only the +significance moves. + +### 6. VERDICT — plain + +**On the pre-registered rule, the pooled verdict is "not distinguishable", and +it is not shipped.** The campaign's primary metric here is round wins/run with a +two-sided exact cross-opponent sign test at p < 0.05, and the pooled number is +**11/14, p = 0.05737** for `arrive` — over the line, by a hair. This is the same +place gate v1 landed in this campaign and the same refusal applies: the verdict +layer is not re-interpreted because the other tests are friendlier. + +For completeness, the same pooled data on the three *other* tests the analyzer +prints all favour the arms, which is why this reads as an **under-powered null at +the MDE boundary** rather than evidence of no effect: + +| test (pooled, `arrive` vs `tfil`) | result | favours | +|---|---|---| +| sign test, wins/run (PRIMARY) | 11/14, **p = 0.05737** | `arrive` — **but over 0.05** | +| sign-flip permutation, wins/run | **p = 0.03113** | `arrive` | +| Wilcoxon, wins/run | **p = 0.03008** | `arrive` | +| 95% CI on Δwins/run | **+0.28, [+0.04, +0.52]** (excludes 0) | `arrive` | +| Δdmg/run | **+6.56** (positive, so no damage cost) | `arrive` | +| MDE, wins/run | 0.31 (observed 0.28) | — | + +**What IS established, cleanly, in both blocks independently:** + +* **The mechanism is real and it is the mechanism the owner described.** The + incoming hit rate falls **18.07% -> 14.92%** (`arrive`), sign test 3/15 + p=0.0352, sign-flip **p=0.0013**, 95% CI **[-5.29, -1.53] pp** excluding 0; + damage taken **-24.57/run** (CI [-39.43, -9.70]). Block 1 gave -3.45 pp / + p=0.0038 and block 2 gave -3.38 pp / p=0.00043 — the tightest, most consistent + result in the whole session, and it is exactly the pathology claim. +* **The outcome direction is positive in every arm in every block**, on both + primary metrics, with **no damage cost** (Δdmg is POSITIVE on all three arms: + +6.56, +6.56, +7.41). +* **`arrive` alone is the strongest arm**, and it is also the simplest: the + hysteresis and the no-reversal speed add nothing measurable on top of it and + the margin arm is the weakest of the three in both blocks. + +**Explicitly NOT claimed:** that this is a proven win, that `TR_MOVEMENT`'s +shipped default should change, or that the pooled p=0.057 is "really" 0.05. The +campaign precedent (`cc11ede` arm D: +20.99 in block 1, +3.36 in block 2) is +exactly why single-block wins here are not promoted. What would settle it is more +OPPONENTS (the unit of evidence), not more runs. + +**The shipped default is untouched.** `TR_MOVEMENT=strafe` remains the default +(`ModularBot_garage/src/ModularBot.nim:118`), `TR_MOVEMENT=tfil` still means +today's tfil, and all three new knobs default to off, so today's behaviour is +reproducible byte-for-byte. + +### 7. Should the owner adopt the knobs in his `.env`? + +**Yes for `TR_TFIL_COMMIT_ARRIVAL=1` and `TR_TFIL_NOREV_SPEED=4`; leave +`TR_TFIL_COMMIT_MARGIN` at 0.** Reasoning, and it is not a wash: + +* `TR_TFIL_COMMIT_ARRIVAL=1` is the load-bearing knob. It is what makes the + target stop flipping under a bot that is still accelerating, it is the best arm + on round wins in both blocks, and it costs nothing: the offline table shows the + pathology count falling 84% and the live hit rate falling 3.15 pp. +* `TR_TFIL_NOREV_SPEED=4` makes the specific thing he watched **impossible rather + than rarer**: with a forward (<=90 deg) safe tile available, a slow mid-flight + switch can no longer take a rearward one (`norevPool`, unit-tested). Offline it + cuts the slow mid-flight reversals a further 102 -> 64 on top of arrival; live + it is neutral-to-slightly-positive (+0.26 wins/run pooled, +0.33 in block 1). + It cannot strand the bot: the pool is never emptied and the all-rearward case + takes the least-bad turn. +* `TR_TFIL_COMMIT_MARGIN=10` is the one to **leave off**. It is a release valve + that SHORTENS holds (mean 24.0 -> 23.7 offline, 391 `hyst` endings), and it is + the weakest arm live in both blocks (+0.19 pooled, +0.25 / +0.13). It is + available if he wants to tune, but there is no evidence for it. + +So the recommended `.env` for his own GUI runs is: + +``` +TR_MOVEMENT=tfil +TR_TFIL_COMMIT_ARRIVAL=1 +TR_TFIL_NOREV_SPEED=4 +``` + +He should expect the dodge to look *smoother and more deliberate* (fewer, longer +commitments) rather than twitchy, and he should see fewer bullets connect. He +should NOT expect a step change in his score from this alone: the measured +outcome effect is +0.28 wins/run with a p of 0.057 on the primary test.