# BitBrain verdict — CLEAN NEGATIVE against real DrussGT **Direct answer (MEASURED).** The BitBrain decay-memory gun does **not** beat the shipped Pattern rack, and it does **not** beat a provably-zero placebo. At 30 runs/arm (210 rounds/arm) `bb_decay` is **+3.3 damage/run** over `control` (permutation p = 0.709, Mann-Whitney p = 0.600) and round wins are **dead level, 97/210 vs 97/210** (p = 1.000). Every one of the twelve pairwise comparisons is a null (all p >= 0.29). The 7-run "shape" that motivated this test — 294 vs 279 damage, 26 vs 22 wins — **did not replicate**: at 30 runs the same difference is +3.3 damage and +0 wins. The one mechanism the offline gate test's diagnosis predicted (a bounded/decaying SBC memory) buys nothing live. **The BitBrain thread closes here**, with a reason instead of an open question. ## What was run (MEASURED) * Frozen from HEAD `795a0e5` via `git archive HEAD`; `ModularBot` binary sha256 `cab6083018672168642fd5a3e62259a260a43ff3ebfc0418dd3cbe15ab8e2005`. * Real DrussGT through the `robocode_shim` bridge, 4 arms x 30 runs x 7 rounds (120 battles, 0 failed), `--conc 7`. * Arms (one env knob each, all four verified live in the boot report): * `control` — no env; shipped `onlyPattern` rack. * `bb_decay` — `TR_RACK_PATTERN=off TR_RACK_BITBRAIN=both TR_BITBRAIN_MEM=decay` (the candidate). * `bb_zero` — as `bb_decay` + `TR_BITBRAIN_RANGE=0`; **this is NOT a zero-shift placebo** (see below), so it is reported as a small-dose treatment. * `bb_placebo` — as `bb_decay` + `TR_BITBRAIN_MIN_OBS=100000000`; the **valid placebo** (the whole BitBrain path runs, but the readout branch is never reached, so the applied shift is provably exactly 0). * `TR_BITBRAIN_LOG=1` on the three BitBrain arms so the applied shift is visible and the placebo can be checked. ## The clear answer to the rubric * `bb_decay` beats `control` on damage/run **AND** round wins with p < 0.05? **No** (+3.3 dmg, p = 0.709; +0 wins, p = 1.000). * `bb_decay` ~= `bb_placebo`? **Yes** (dmg p = 0.897, wins p = 0.918). So any difference the BitBrain path makes is **plumbing/noise, not learning** — and here it makes no difference at all. * `bb_zero` ~= `control` **and** `bb_decay` > both? **Not the situation**: `bb_zero` is not a zero-shift placebo, and `bb_decay` beats nothing. * Nothing separates -> **CLEAN NEGATIVE.** Not spun: this is a successful outcome that closes the thread. ## The placebo: `TR_BITBRAIN_RANGE=0` is clamped, so it is NOT a zero-shift arm (MEASURED) `common_libs/guns/bitbrain_gun.nim:207`: ```nim result.maxDeg = clamp(envFloatBB(BB_RANGE_ENV, BB_RANGE_DEF), 1.0, 180.0) ``` `TR_BITBRAIN_RANGE=0` is therefore clamped to `1.0` degree, and the class centres become +/-0.97 deg, not 0. Confirmed live: the boot report shows `[env] TR_BITBRAIN_RANGE = 1.0 (source: env)` and the `[bb]` log emits `shift=-1.0deg` / `shift=+1.0deg` (edge classes 0 and 31). So `bb_zero` applies a systematic ~1 deg correction and is **not** the intended placebo. Per the task's fallback this arm was kept only as a small-dose treatment; the true placebo is `bb_placebo`, which emits **0 `[bb]` lines in 30/30 runs** — a zero applied shift by construction (`bbLog` is reached only inside the same `trained >= minObs` branch that computes the shift, so a never-firing readout is silent). Nominally `bb_zero` trends *worse* (-5.5 dmg/run vs control, p = 0.51; -7.7 vs the true placebo, p = 0.35), consistent with a tiny systematic mis-aim rather than a neutral probe — and still inside the noise. ## Raw analyzer output (verbatim, `tools/ab/ab_analyze.py`) ``` # session /tmp/ab/bbverdict # commit=795a0e59febb4d9b9722130ad47cd2dd8f72a315 binary_sha256=cab6083018672168642fd5a3e62259a260a43ff3ebfc0418dd3cbe15ab8e2005 rounds=7 runs=30 conc=7 ts=2026-09-24T22:37:38+02:00 ARM SUMMARY arm runs dmg/run dmgtk/run wins win% shots/run hitstk/run -------------------------------------------------------------------------- control 30 280 206 97/210 46.2 796 92.5 bb_decay 30 284 206 97/210 46.2 797 93.6 bb_zero 30 275 209 93/210 44.3 781 91.8 bb_placebo 30 283 209 95/210 45.2 786 92.2 PER-RUN (never just the mean) control dmg: r1=293 r2=296 r3=352 r4=263 r5=282 r6=292 r7=246 r8=271 r9=246 r10=261 r11=263 r12=175 r13=302 r14=288 r15=280 r16=333 r17=244 r18=293 r19=317 r20=249 r21=283 r22=296 r23=278 r24=281 r25=272 r26=292 r27=238 r28=332 r29=295 r30=299 wins: r1=3/7 r2=5/7 r3=4/7 r4=3/7 r5=5/7 r6=3/7 r7=2/7 r8=2/7 r9=3/7 r10=3/7 r11=3/7 r12=0/7 r13=4/7 r14=4/7 r15=3/7 r16=2/7 r17=4/7 r18=3/7 r19=3/7 r20=3/7 r21=2/7 r22=2/7 r23=4/7 r24=5/7 r25=4/7 r26=3/7 r27=3/7 r28=6/7 r29=3/7 r30=3/7 bb_decay dmg: r1=284 r2=247 r3=273 r4=300 r5=241 r6=248 r7=276 r8=299 r9=227 r10=279 r11=305 r12=349 r13=302 r14=318 r15=279 r16=244 r17=307 r18=293 r19=328 r20=212 r21=296 r22=296 r23=362 r24=258 r25=245 r26=298 r27=308 r28=294 r29=285 r30=259 wins: r1=4/7 r2=2/7 r3=5/7 r4=3/7 r5=3/7 r6=3/7 r7=3/7 r8=3/7 r9=2/7 r10=4/7 r11=2/7 r12=4/7 r13=3/7 r14=6/7 r15=3/7 r16=2/7 r17=5/7 r18=4/7 r19=4/7 r20=2/7 r21=2/7 r22=2/7 r23=6/7 r24=1/7 r25=3/7 r26=3/7 r27=6/7 r28=4/7 r29=1/7 r30=2/7 bb_zero dmg: r1=248 r2=295 r3=267 r4=310 r5=277 r6=313 r7=264 r8=276 r9=277 r10=236 r11=302 r12=262 r13=315 r14=251 r15=314 r16=297 r17=231 r18=266 r19=212 r20=224 r21=249 r22=235 r23=314 r24=314 r25=285 r26=262 r27=296 r28=318 r29=271 r30=265 wins: r1=1/7 r2=4/7 r3=3/7 r4=2/7 r5=3/7 r6=6/7 r7=3/7 r8=4/7 r9=3/7 r10=3/7 r11=3/7 r12=2/7 r13=4/7 r14=4/7 r15=4/7 r16=2/7 r17=1/7 r18=4/7 r19=3/7 r20=2/7 r21=3/7 r22=2/7 r23=5/7 r24=2/7 r25=4/7 r26=2/7 r27=4/7 r28=3/7 r29=3/7 r30=4/7 bb_placebo dmg: r1=279 r2=285 r3=328 r4=253 r5=275 r6=286 r7=348 r8=256 r9=258 r10=303 r11=278 r12=242 r13=291 r14=234 r15=341 r16=274 r17=240 r18=267 r19=312 r20=228 r21=302 r22=337 r23=320 r24=266 r25=308 r26=291 r27=255 r28=275 r29=308 r30=234 wins: r1=2/7 r2=3/7 r3=4/7 r4=1/7 r5=4/7 r6=2/7 r7=4/7 r8=2/7 r9=2/7 r10=4/7 r11=4/7 r12=2/7 r13=5/7 r14=2/7 r15=4/7 r16=3/7 r17=3/7 r18=3/7 r19=4/7 r20=2/7 r21=3/7 r22=4/7 r23=4/7 r24=4/7 r25=5/7 r26=2/7 r27=3/7 r28=3/7 r29=5/7 r30=2/7 PAIRWISE PERMUTATION TEST (per-run values) + MANN-WHITNEY CROSS-CHECK permutation: exact when C(n,na) <= 20,000,000; otherwise Monte-Carlo 1,000,000 draws, seed=0x5eed5eed, p = (cnt+1)/(B+1), se = sqrt(p(1-p)/(B+1)) metric A B diff(A-B) perm p method MC se MW p MW U ------------------------------------------------------------------------------------------------- dmg/run control bb_decay -3.313 0.7091 MC/B=1,000,000 0.0005 0.5997 414.0 round wins control bb_decay +0.000 1.0000 MC/B=1,000,000 0.0000 0.7468 428.5 dmg/run control bb_zero +5.551 0.5086 MC/B=1,000,000 0.0005 0.5493 409.0 round wins control bb_zero +0.133 0.7369 MC/B=1,000,000 0.0004 0.6535 420.5 dmg/run control bb_placebo -2.175 0.8030 MC/B=1,000,000 0.0004 0.9117 442.0 round wins control bb_placebo +0.067 0.9091 MC/B=1,000,000 0.0003 0.8356 436.0 dmg/run bb_decay bb_zero +8.863 0.2947 MC/B=1,000,000 0.0005 0.4376 397.0 round wins bb_decay bb_zero +0.133 0.7597 MC/B=1,000,000 0.0004 0.9027 441.5 dmg/run bb_decay bb_placebo +1.138 0.8969 MC/B=1,000,000 0.0003 0.7845 431.0 round wins bb_decay bb_placebo +0.067 0.9178 MC/B=1,000,000 0.0003 0.9513 445.5 dmg/run bb_zero bb_placebo -7.725 0.3525 MC/B=1,000,000 0.0005 0.4733 401.0 round wins bb_zero bb_placebo -0.067 0.9072 MC/B=1,000,000 0.0003 0.7763 431.0 MINIMUM DETECTABLE EFFECT (two-sample, alpha=0.05 two-sided, 80% power; MDE = 2.8016*sd*sqrt(2/n)) metric n/arm sd(control) MDE(abs) MDE vs control mean ---------------------------------------------------------------- dmg/run 30 33.669 24.355 8.7% of 280.4 round wins 30 1.165 0.843 26.1% of 3.2 ROUND-LEVEL TEST (pooled rounds, Fisher exact) vs `control` — ANTI-CONSERVATIVE: rounds cluster within runs arm ref wins arm wins p ---------------------------------------------- bb_decay 97/210 97/210 1.0000 bb_zero 97/210 93/210 0.7687 bb_placebo 97/210 95/210 0.9220 LIVENESS (arm env applied in the bot's own boot report) control OK (30/30 runs: no arm env; report present) bb_decay OK (30/30 runs: TR_RACK_PATTERN=off TR_RACK_BITBRAIN=both TR_BITBRAIN_MEM=decay TR_BITBRAIN_LOG=1 applied) bb_zero OK (30/30 runs: TR_RACK_PATTERN=off TR_RACK_BITBRAIN=both TR_BITBRAIN_MEM=decay TR_BITBRAIN_RANGE=0 TR_BITBRAIN_LOG=1 applied) bb_placebo OK (30/30 runs: TR_RACK_PATTERN=off TR_RACK_BITBRAIN=both TR_BITBRAIN_MEM=decay TR_BITBRAIN_MIN_OBS=100000000 TR_BITBRAIN_LOG=1 applied) [bb] APPLIED-SHIFT CHECK (from bot stdout; needs TR_BITBRAIN_LOG=1). A provably-zero placebo emits ZERO [bb] lines. arm runs w/log lines min max zeros control 0/30 0 - - - bb_decay 30/30 905 -38.80 +36.20 0 bb_zero 30/30 408 -1.00 +1.00 9 bb_placebo 0/30 0 - - - ROUND-WIN ATTRIBUTION (events primary; score tie-break for mutual-kill / timeout rounds) control wins==firstPlaces 30/30 runs OK; single-death rounds agree with score 205/205 (5 tie-broken) bb_decay wins==firstPlaces 30/30 runs OK; single-death rounds agree with score 209/209 (1 tie-broken) bb_zero wins==firstPlaces 30/30 runs OK; single-death rounds agree with score 209/209 (1 tie-broken) bb_placebo wins==firstPlaces 30/30 runs OK; single-death rounds agree with score 207/207 (3 tie-broken) ``` ## Minimum detectable effect (MEASURED) — what this test CAN and CANNOT see From the observed control-arm per-run SD (damage SD = 33.7; wins SD = 1.17) at 30 runs/arm, alpha = 0.05 two-sided, 80% power: * **damage/run: MDE = 24.4** (8.7% of the 280 control mean). * **round wins: MDE = 0.84 wins/run** (26.1% of the 3.2 mean). So this test rules out a `bb_decay` damage gain of **>= 24 dmg/run** (and a win gain of **>= 0.84/run**). A real effect of, say, +10-15 dmg/run — the size the 7-run result hinted at — **would not be detectable here**, and is **not** ruled out by the null. That is the honest bound: "no effect >= 24 dmg/run", not "no effect". The placebo isolates the mechanism more sharply: `bb_decay` vs `bb_placebo` is +1.1 dmg/run (p = 0.897) and +0.07 wins (p = 0.918), so the learning-specific effect is bounded by the same ~24 dmg/run and shows no sign. ## Why the earlier 7-run shape was misleading (MEASURED) | metric | 7 runs/arm (commit `795a0e5` session) | 30 runs/arm (this session) | |---|---|---| | `control` dmg/run | 279 | 280 | | `bb_decay` dmg/run | 294 (+15) | 284 (+3.3) | | `control` round wins | 22/49 | 97/210 | | `bb_decay` round wins | 26/49 (+4) | 97/210 (+0) | The 7-run difference was inside the noise; it has shrunk to zero, not grown. ## MEASURED vs INFERRED * **MEASURED:** the damage/win table, the pairwise permutation p-values (Monte-Carlo, 1,000,000 draws, seed `0x5eed5eed`, reported with MC SE) and the Mann-Whitney cross-check, the MDE from the observed SD, the liveness lines, the `[bb]` shift check, and the `TR_BITBRAIN_RANGE` clamp (`= 1.0` in the live boot report). * **INFERRED:** that a true effect below ~24 dmg/run would need more runs or a lower-variance opponent to resolve. The offline gate-test diagnosis (decay memory should help) is **not** supported live; whether a *different* memory regime would help is **not** tested here (only `decay` was). ## Reproduce ```sh tools/ab/ab_run.sh --arms /tmp/ab/arms_bbverdict.txt --runs 30 \ --outdir /tmp/ab/bbverdict --conc 7 python3 tools/ab/ab_analyze.py /tmp/ab/bbverdict ``` Analyzer tooling improved in the same change: exact enumeration is kept when `C(n, na) <= 20e6` (7v7), otherwise a seeded Monte-Carlo permutation test (`MC_DRAWS = 1,000,000`, `MC_SEED = 0x5eed5eed`) with its standard error, plus a tie-corrected Mann-Whitney U cross-check and the MDE line. See `tools/ab/README.md`.