# Is BitBrain actually better than Pattern in melee?

**This is the repository's FIRST melee measurement.** Every A/B before this one
was 1v1 vs DrussGT. This doc answers one question with a real number:

> *"btw bitbrain gun is crushing in melee, the fast adaptation is a killer
> feature there"* — the owner.

## TL;DR (direct answer)

**Not detectable at this sample size.** On the metric that matters most in a
melee (the server's round score = damage + survival bonus), the three BitBrain
arms land within ±62 points of the shipped Pattern rack, and every difference is
well inside the noise: permutation **p = 0.16 – 0.71**, all **< the MDE of ±151
score points (~5.1 %)**. Placement is saturated: every arm finishes **rank 1**
(ModularBot's TFIL movement beats this field regardless of gun). Round wins
*hint* in BitBrain's favour — **112/112** (retained, learned) and **111/112**
(perRound) vs Pattern's **109/112** — but that +0.19 rounds/run is not
significant (**perm p = 0.225, Mann-Whitney p = 0.080**) and is half the
**MDE of 0.40 rounds/run**. So:

| claim | verdict |
|---|---|
| BitBrain beats Pattern on **round wins** in melee | **hint only, not significant** (p=0.225; MDE=0.40 wins/run) |
| BitBrain beats Pattern on **survival** in melee | **not detectable** (p=0.23–1.00) |
| BitBrain beats Pattern on **placement** in melee | **no — every arm sweeps** (rank 1 everywhere) |
| BitBrain beats Pattern on **score/damage** in melee | **no — not detectable** (p=0.16–0.71; all diffs << MDE) |

**MEASURED** (this doc): the numbers above. **INFERRED** (not measured): the
mechanism (Pattern loses history across a target switch; BitBrain resets and
re-adapts). The mechanism is real and was exercised — see the target-switching
liveness — but at this sample it buys no detectable score.

---

## Why melee is a different question

Pattern's strength comes from accumulating history with **one** enemy. Melee
rotates targets, so per-enemy histories fragment; a gun that **resets per target
and re-adapts fast** should gain exactly where Pattern loses. BitBrain is built
for that: `TR_BITBRAIN_RESET_ON_TARGET` defaults to **on**, and its memory modes
(`perRound` / `retained` / `decay`) exist for changing targets. That is a
plausible mechanism — but a plausible mechanism is not a measurement, and
**melee is higher-variance than 1v1** (where a 6-4 result already had P=0.353),
so an impression is worth even less here. This is the first real number.

## Melee scoring caveat

Melee is scored differently from 1v1 and the framework does **not** expose raw
damage. All numbers below come from the framework's own fields
(`BattleResult` / `BotResult` / `BotRoundResult`):

* **`firstPlaces`** — rounds won (rank 1 in a round). The run's primary outcome.
* **`survivalCount`** — total ticks survived across the run's 7 rounds.
* **`rank` / round `rank`** — final placement and per-round placement (1 = best).
* **`totalScore`** — the **server's round score**, which is **damage dealt plus
  the survival bonus**, *not raw damage*. It is the closest available proxy for
  damage; treat it as "score", not "damage". `score share` = ModularBot's score
  ÷ (all four bots' scores).

## Method

*Field (fixed across arms):* 4-bot Free-For-All — **ModularBot + WaveSurfer +
PatternMover + RandomMover**. Mixed by design: WaveSurfer is the strong reactive
mover (linear gun), PatternMover the deterministic pattern (Pattern's home turf),
RandomMover the erratic one. All three shoot.

*Arms (only ModularBot's gun rack differs):*

| arm | env |
|---|---|
| `pattern` | *(no env — shipped default: Pattern-only rack)* |
| `bb_round` | `TR_RACK_PATTERN=off TR_RACK_BITBRAIN=melee TR_BITBRAIN_MEM=perRound TR_BITBRAIN_LOG=1` |
| `bb_ret` | `TR_RACK_PATTERN=off TR_RACK_BITBRAIN=melee TR_BITBRAIN_MEM=retained TR_BITBRAIN_LOG=1` |
| `bb_learn` | `TR_RACK_PATTERN=off TR_RACK_BITBRAIN=melee TR_BITBRAIN_MEM=decay TR_BITBRAIN_GAINS=1.0,1.25,1.5,2.0 TR_BITBRAIN_LOG=1` |

*Protocol:* **16 independent runs × 7 rounds per arm = 112 rounds/arm.** Each
run is its own battle with fresh random initial positions (server
`--enable-initial-position`); runs are not paired. A dirty tree cannot leak in:
the frozen ModularBot is built with `git archive HEAD`.

```
commit        = 0f5cfe37b24952ae07fee0a7db526317ae127e6c
binary sha256 = 11bba2735413bc35ab3b61ead711c51d2223183ad9da510799bc7d76b99b22ca
movement      = tfil (shipped default) in every run — the A/B isolates the gun rack
n             = 16 runs/arm, 7 rounds/run
```

`TR_RACK_BITBRAIN=melee` was verified as *melee-rack-only* (not `both`): the
boot line reads `[env] rack active 1v1 = FULL` and
`[env] rack active melee = BITBRAIN`, i.e. BitBrain is admitted **only** in the
melee rack. The `pattern` arm reads `[env] rack active melee = PATTERN`.

## Results

### Arm summary

```
arm        runs   wins/rd    score    surv  rank  meanrank  score share  targets  tchanges
bb_learn     16   112/112     2945    1050  1.00     1.000        60.2%     3.00     66.75
bb_ret       16   112/112     2903    1047  1.00     1.000        59.4%     3.00     68.06
bb_round     16   111/112     2998    1034  1.00     1.009        60.8%     3.00     68.75
pattern      16   109/112     2965    1031  1.00     1.000        59.5%     3.00     66.44
```

`score` / `surv` are per-run means; `wins/rd` is pooled round wins.

### Per-run values (never just the mean)

```
bb_learn  wins:  r1=7 r10=7 r11=7 r12=7 r13=7 r14=7 r15=7 r16=7 r2=7 r3=7 r4=7 r5=7 r6=7 r7=7 r8=7 r9=7
          score: r1=2883 r10=2757 r11=2906 r12=3145 r13=2911 r14=3068 r15=2758 r16=2715 r2=3107 r3=2848 r4=3190 r5=2912 r6=2859 r7=3092 r8=2907 r9=3066
          surv:  r1=1050 … (all 1050)
          mrank: all 1.00
bb_ret    wins:  r1=7 r10=7 r11=7 r12=7 r13=7 r14=7 r15=7 r16=7 r2=7 r3=7 r4=7 r5=7 r6=7 r7=7 r8=7 r9=7
          score: r1=2891 r10=2770 r11=2792 r12=2934 r13=2943 r14=2890 r15=2817 r16=2999 r2=2994 r3=2913 r4=2920 r5=2890 r6=3011 r7=2945 r8=2943 r9=2802
          surv:  r5=1000, all others 1050
          mrank: all 1.00
bb_round  wins:  r1=7 r10=6 r11=7 r12=7 r13=7 r14=7 r15=7 r16=7 r2=7 r3=7 r4=7 r5=7 r6=7 r7=7 r8=7 r9=7
          score: r1=3040 r10=2944 r11=3125 r12=3110 r13=3037 r14=2993 r15=2985 r16=2842 r2=2835 r3=3045 r4=3180 r5=2952 r6=3126 r7=2927 r8=3021 r9=2804
          surv:  r10=900, r9=950, all others 1050
          mrank: r10=1.14, all others 1.00
pattern   wins:  r1=7 r10=7 r11=7 r12=7 r13=7 r14=7 r15=7 r16=7 r2=7 r3=6 r4=7 r5=6 r6=7 r7=6 r8=7 r9=7
          score: r1=3082 r10=2795 r11=2897 r12=2915 r13=3176 r14=2922 r15=2860 r16=2878 r2=3261 r3=2957 r4=3106 r5=2869 r6=3194 r7=2728 r8=2869 r9=2931
          surv:  r3=1000, r7=900, r14=950, all others 1050
          mrank: all 1.00
```

The only structural difference is that Pattern lost three rounds (r3, r5, r7) and
`bb_round` one (r10); every other round is a ModularBot sweep. `surv` < 1050
marks the runs where ModularBot died at least once. The raw per-run JSON for all
64 runs is committed at
`common_libs/tests/fixtures/melee_bitbrain_ab/<arm>/run<N>.json`.

### Statistics (per-run, two-sided)

Permutation test (Monte-Carlo, 1,000,000 draws, seed `0x5eed5eed`, `p=(cnt+1)/(B+1)`)
and Mann-Whitney U cross-check, reused verbatim from `tools/ab/ab_analyze.py`.
`diff(A-B)` is `pattern − bitbrain`.

```
metric       A        B          diff(A-B)   perm p   MW p
score        pattern  bb_learn     +19.75     0.7117   0.6109
score        pattern  bb_ret       +61.63     0.1594   0.5590
score        pattern  bb_round     -32.88     0.4900   0.3365
survival     pattern  bb_learn     -18.75     0.2254   0.0800
survival     pattern  bb_ret       -15.63     0.3507   0.2791
survival     pattern  bb_round      -3.13     1.0000   0.6982
wins         pattern  bb_learn      -0.188     0.2251   0.0795
wins         pattern  bb_ret        -0.188     0.2251   0.0795
wins         pattern  bb_round      -0.125     0.5984   0.3080
mean_rank    pattern  bb_*          ±0.000     1.0000   0.35–1.00
rank         pattern  bb_*          ±0.000     1.0000   1.0000
score_share  pattern  bb_learn      -0.007     0.5082   0.6109
score_share  pattern  bb_ret        +0.001     0.9046   0.8653
score_share  pattern  bb_round      -0.013     0.2908   0.2662
```

### Minimum detectable effect (α=0.05 two-sided, 80 % power)

`MDE = 2.8016 · sd(pattern) · sqrt(2/n)`, n=16/arm.

```
metric      sd(ref)    MDE      ref mean    MDE as % of mean
score       152.85    151.40    2965.0           5.1 %
survival     44.25     43.83    1031.3           4.3 %
wins          0.40      0.40       6.81           5.9 %
mean_rank     0.00      0.00       1.000          —
```

Every observed effect is **below its MDE**, so this is a *powered null*, not an
under-powered one — at least for effects ≥ ~5 % of the score.

## Liveness — the melee premise WAS exercised

Target switching is the whole mechanism, so it had to be proven, not assumed.

* **Arm applied:** the per-run boot report shows the rack actually flipped —
  `pattern`: `[env] rack active melee = PATTERN`; `bb_*`:
  `[env] rack active melee = BITBRAIN` with `TR_RACK_PATTERN=off`,
  `TR_RACK_BITBRAIN=melee`. All 64 runs passed the liveness check (16/16 per arm).
* **Movement held fixed:** `[env] TR_MOVEMENT = tfil (source: default)` in every
  run, so the difference is the gun rack alone.
* **Every run faced all three enemies and switched targets repeatedly.** From
  the bot's own `[config] … target=#N` lines: **3.00 distinct targets/run** and
  **66–69 target changes/run** (about 10 per round). Example transitions from one
  run: `#1 → #4 → #2 → #4 → #1 → #2 → …`.
* **BitBrain really reset on every switch.** `[bb-reset] reason=target_change`
  fires once per target change: **66.75** (perRound), **68.19** (retained),
  **66.75** (decay) per run — equal, within rounding, to the target-change count.
  Example `[bb]` line from a live run:
  `[bb] t=440 band=450.+ gain=0.25 shift=-3.39deg rate=0.818 n=11. ncand=5 trained=11 pend=29 dropped=95 mode=perRound`.
  The `pattern` arm logs **zero** `[bb]`/`[bb-reset]` lines (BitBrain never
  spawned), confirming the arms are cleanly separated.

**So the premise held** (targets rotated, the gun reset), but the fast adaptation
bought no measurable edge over Pattern's per-round wipe in this field.

## Direct answer

At **16 runs/arm (112 rounds/arm)**, in a 4-bot melee against
WaveSurfer + PatternMover + RandomMover:

* **Round wins:** BitBrain is *slightly* ahead (112/112 and 111/112 vs 109/112),
  but the difference is **not significant** (perm p=0.225; MW p=0.080) and is
  below the 0.40-wins/run MDE. Do **not** claim a win-rate gain from this.
* **Survival:** **not detectable** (p=0.23–1.00).
* **Placement:** **no difference** — ModularBot is rank 1 in every arm.
* **Score/damage:** **not detectable** — differences of −63 to +33 points
  (p=0.16–0.71) against an MDE of ±151 (≈5 %).

The mechanism is plausible and was demonstrably exercised, but the owner's claim
that BitBrain is "crushing in melee" is **not supported by this measurement**.
The honest reading is the opposite of spin in either direction: **the first
melee number is a null.**

### Why the null is weak evidence in one direction

The field is not discriminating enough to separate the guns on *outcome*:
ModularBot's TFIL movement wins 97–100 % of rounds no matter which gun is in the
rack, so wins/rank hit a ceiling. Only the score margin can move, and its MDE is
~5 %. A smaller true gun effect (<5 % score) would be invisible here; a larger
one would not.

## Reproduce

```sh
# build the runner (once)
nim c --nimcache:/tmp/nc_j116 --path:common_libs \
  common_libs/tests/measure_melee_bitbrain_ab.nim

# run the session (needs TR_SERVER_JAR + the TR runner jar; ~15–25 min under load)
MELEE_RUNS=16 MELEE_ROUNDS=7 \
  common_libs/tests/run_melee_bitbrain_ab.sh /tmp/melee_bitbrain_ab

# analyze the committed fixtures (no Java needed)
python3 common_libs/tests/analyze_melee_ab.py \
  common_libs/tests/fixtures/melee_bitbrain_ab
```

## Files

* `common_libs/tests/measure_melee_bitbrain_ab.nim` — the runner (4-bot melee,
  per-arm env, per-run liveness/JSON, binary built from `git archive HEAD`).
* `common_libs/tests/run_melee_bitbrain_ab.sh` — parallel arm driver.
* `common_libs/tests/analyze_melee_ab.py` — per-run table + permutation + MW +
  MDE (reuses `tools/ab/ab_analyze.py`'s stats functions).
* `common_libs/tests/fixtures/melee_bitbrain_ab/` — the 64 committed per-run
  JSONs + `analysis.json` + `analysis_report.txt`.

## MEASURED vs INFERRED

**MEASURED:** commit `0f5cfe3`, binary `11bba27`; 4-bot melee, 4 arms × 16 runs
× 7 rounds; the wins/survival/rank/score means and per-run values; the
permutation p-values, Mann-Whitney p-values and MDEs; the rack boot lines, the
3 distinct targets/run, 66–69 target changes/run and the per-switch `[bb-reset]`.
All of the above are reproducible from the committed per-run JSON.

**INFERRED:** that Pattern's history fragmentation is *the* reason to expect a
melee gain; that `perRound`/`retained`/`decay` differ in adaptation speed in a
way this test could resolve; that a stronger field (or a >5 % effect) is where a
difference would show. None of these are claimed as measured.