SirRoboGarage/docs/bitbrain_campaign.md

# BitBrain campaign ledger

**Goal:** make ModularBot's gun **beat Pattern live** against the real DrussGT.
The user has granted full freedom over the gun ("change input, output, every
knob of it") and accepts it may fail — the deliverable is that the attempt is
visible and evidence-backed.

**THE FINAL VERDICT IS ALWAYS LIVE.** Everything in this file except the
`## Phase N` verdict lines is offline, open-loop, on a *fixed recorded enemy
trajectory*. Per `docs/offline_harness_trust.md` (commit `e40c849`) the offline
harness is trustworthy for exactly one thing: **per-gun single-tick prediction
quality on a fixed enemy trajectory** — and it is *never* trustworthy for
closed-loop questions (movement, range, round length, adaptation, gun
selection, damage, wins, survival). No offline number here is a win/damage
claim, and no phase may be called a success without a live A/B
(`tools/ab/ab_run.sh`, server-side event hit rate, left-running).

Every claim below is tagged **[MEASURED]** (a command in §0 reproduces it) or
**[INFERRED]** (reasoning from measured facts).

---

## Phase 0 — BUILD THE RULER AND ESTABLISH THE BAR  *(owner: overnight job, committed)*

### 0.1 The ruler

`common_libs/gun_harness/prediction_quality.nim` + `common_libs/tests/run_prediction_quality.nim`.

At each recorded tick the shooter sits at `O = (selfX, selfY)`. For a bullet of
speed `v` the **true interception point** is the first fractional time `t > 0`
at which the enemy's ACTUAL recorded track reaches distance `v*t` from `O`
(linear interpolation between recorded ticks). A bullet fired along the bearing
to `E(t)` coincides with the enemy at `t`. Angular error is
`wrap180(bearing(O→pred) − bearing(O→E(t)))` in **degrees**; every tick is
scored for the four power bins (speeds 17/15.5/14/11), all bands share that
horizon set. Per range band we report `mean|err|`, RMSE, mean signed err and the
hit-probability proxy `mean(|err| ≤ atan(18/range))`.

The integer-tick solve from `analyze_lead_capture_by_range.py` (commit
`f91e121`) is kept as `interceptBearingQuant` and reported as `OracleQuant`; the
ruler ships the **continuous** solve because it separates recorded hits from
misses slightly better and removes the coarse solve's own overshoot
(§0.3.6). `--ruler quant` selects the integer solve.

**Data:** the recorded live-vs-real-DrussGT corpus `/tmp/tfil_ab2/out`
(70 battles / 490 rounds / 899 607 ticks + `.events.jsonl` + `.rounds.json`),
**verified present before use**. It lives in `/tmp` and is therefore ephemeral;
if a later job finds it gone, regenerate it with the A/B harness
(`tools/ab/ab_run.sh`, which sets `TR_RECORD_WORLDSTATE` so ModularBot appends
per-tick world state) and point `--corpus` at the new output root. Layout:
`<root>/<arm>/runN.jsonl` + `runN.events.jsonl` + `runN.jsonl.rounds.json`.
149 MB of JSONL is converted once per run into a compact float32 `.qcache`
(keyed on source mtime+size) and ALL measurement is taken from the cache, so two
runs are byte-identical. See §0.4 for speed.

### 0.2 Validation — the ruler must pass ALL of these **[MEASURED]**

Run: `nim c -d:release --nimcache:/tmp/nc_j98 -r common_libs/tests/run_prediction_quality.nim`

**1. Recorded HITS separate from recorded MISSES** (our ACTUAL server-fired
bearings, scored against the SAME interception solve):

| ruler | hits n | hits mean\|err\| | misses n | misses mean\|err\| | separation |
|---|---|---|---|---|---|
| continuous | 5480 | **1.360° / 10.5 px** | 48304 | **16.597° / 140.3 px** | **12.20× deg / 13.34× px** |
| integer | 5480 | 1.478° / 11.4 px | 48304 | 16.724° / 141.3 px | 11.32× / 12.43× |

(The earlier validated run quoted 11.59× / 11.6 px on hits; reproduced and
improved.) The continuous ruler is shipped because it separates better.

**2. Perfect oracle scores 0.** Max `|err|` over all 3 598 428 tick-bins =
**0.000000°**. OK.

**3. A static line-of-sight gun is far from the predictor on learnable motion.**
On a synthetic constant-velocity and a seeded random-walk trajectory the
ordering is exactly as physics demands: HeadOn (zero lead) is the worst, Pattern
and naive-linear are near-zero, and the lead-gain arms overshoot monotonically.
On the real DrussGT corpus the static gun is *not* worst — see §0.3.4, this is a
genuine property of the corpus, not a harness defect.

**4. Determinism.** Two full 70-run sweeps, stdout diffed with the two wall-time
lines excluded: **byte-identical**. (The only difference between the two raw
outputs is `wall time 406.88s` vs `402.41s` and the derived ms-per-tick-bin.)
**[MEASURED]**

**5. A real bug was found and fixed by this validation.** The ruler's
`wrap180` used Nim's float `mod`, which keeps the dividend's sign (C `fmod`), so
`(x+180) mod 360 − 180` returned `x−360` instead of the wrapped equivalent for
`x < −180`. This inflated the negative tail of every error (maxAbs read ~360°
instead of ~180°) and made HeadOn's mean error disagree with `mean|required|`.
After the fix HeadOn's `mean|err|` equals `mean|required lead|` to the last
digit at every band (see the `HO |err| / HO |req| / Pat|req|` columns in the
fixture). **A wrong ruler is worse than no ruler; this was the most important
10 minutes of the phase.**

### 0.3 THE BAR — per-band numbers (70 runs, 3 598 428 tick-bins) **[MEASURED]**

Format: `mean|err| deg` and, in brackets, `hitProxy`. `hitProxy` is the fraction
of tick-bins aimed within `atan(18/range)` of the true interception point.

| band | Pattern | naive-linear | TMHorizon | BitBrain | HeadOn (static) | Oracle |
|---|---|---|---|---|---|---|
| 0–100 | **10.56** [0.699] | 17.92 [0.651] | 10.68 [0.696] | 11.12 [0.681] | 19.62 [0.342] | 0.00 [1.000] |
| 100–200 | **14.75** [0.342] | 14.63 [0.388] | 14.84 [0.342] | 15.11 [0.324] | 19.98 [0.172] | 0.00 [1.000] |
| 200–300 | **16.61** [0.185] | 17.57 [0.192] | 16.64 [0.179] | 16.84 [0.175] | 17.34 [0.133] | 0.00 [1.000] |
| 300–450 | 17.53 [0.104] | 20.98 [0.100] | 17.57 [0.100] | 17.58 [0.103] | **14.61** [0.105] | 0.00 [1.000] |
| 450+ | 16.19 [0.077] | 22.86 [0.054] | 16.20 [0.076] | 16.20 [0.077] | **12.33** [0.098] | 0.00 [1.000] |

`n`: 4 423 / 24 908 / 74 215 / 1 119 777 / 2 311 323. Skipped (no valid
interception): 63 782 tick-bins (≈1.7 %).

**0.3.1 DIRECT ANSWER — the gap between Pattern and the oracle ceiling:**

| band | Pattern hitProxy | Oracle hitProxy | headroom (pp) |
|---|---|---|---|
| 0–100 | 0.6993 | 1.0000 | **+30.07** |
| 100–200 | 0.3418 | 1.0000 | **+65.82** |
| 200–300 | 0.1850 | 1.0000 | **+81.50** |
| 300–450 | 0.1036 | 1.0000 | **+89.64** |
| 450+ | 0.0767 | 1.0000 | **+92.33** |

**[INFERRED, important]** The oracle is *non-causal*: it aims with perfect
knowledge of the enemy's future, so its 100 % is a definition, not an
achievement, and the 92 pp at 450+ is an UPPER bound that contains both "a
better predictor could get this" and "this is physically unknowable". The
**realistic** causal bound measured today is the best arm at 450+: **HeadOn at
9.8 %**, barely above Pattern's 7.7 %. So the campaign is playing for a few
percentage points at long range, not for 92 pp. The honest target statement is
"raise the 450+ proxy from 7.7 % toward the ~10 % causal band", not "toward
100 %".

**0.3.2 The lead-gain sweep on Pattern is a DEAD END [MEASURED].** Multiply
Pattern's angular lead over LOS by a constant, per band:

| band | gain 1.0 | gain 1.5 | gain 2.0 | gain 3.0 |
|---|---|---|---|---|
| 0–100 | **10.56** | 13.76 | 20.11 | 34.77 |
| 100–200 | **14.75** | 19.64 | 26.78 | 42.97 |
| 200–300 | **16.61** | 22.26 | 29.47 | 45.23 |
| 300–450 | **17.53** | 23.28 | 29.96 | 44.27 |
| 450+ | **16.19** | 21.25 | 27.00 | 39.27 |

Gain 1.0 wins at EVERY band. Scaling Pattern's lead up makes it strictly worse.
This is the single most important negative result of Phase 0 and it should stop
any later job from "just adding more lead".

**0.3.3 The naive-linear / capture tension, resolved [MEASURED].** Capture slope
= regression of the arm's own lead on the required lead (job-95's statistic);
corr = Pearson correlation of the arm's lead with the required lead. **corr is
the informative number; a large slope on an uncorrelated lead is just amplified
noise.**

| band | mean\|req\| | Pattern cap / corr | naive-linear cap / corr | TMHorizon | BitBrain |
|---|---|---|---|---|---|
| 100–200 | 19.98 | 0.553 / 0.612 | 0.592 / 0.523 | 0.560 / 0.614 | 0.570 / 0.611 |
| 300–450 | 14.61 | 0.278 / 0.266 | 0.476 / 0.324 | 0.273 / 0.262 | 0.280 / 0.266 |
| 450+ | 12.33 | 0.175 / 0.165 | 0.310 / 0.178 | 0.174 / 0.164 | 0.175 / 0.165 |

Yes — on this corpus the naive-linear predictor applies **~1.8× more lead** than
Pattern at 450+ (0.310 vs 0.175; job-95 measured ~2×). Job-95's "we under-lead"
reading is confirmed. **But** the two arms carry almost the same lead
*information* (corr 0.178 vs 0.165), so the extra amplitude buys nothing and
costs angular accuracy: naive-linear's `mean|err|` is 22.86° vs Pattern's
16.19° at 450+. **Conclusion: the campaign's lever is lead INFORMATION
(correlation), not lead RESPONSE (capture slope).** Capturing more of an
uninformative lead is worse than capturing little of it — which is also exactly
why the gain sweep fails.

**0.3.4 The surprise: at long range, static line-of-sight beats Pattern.**
HeadOn (aim at the enemy's current position) has `mean|err|` 14.61°/12.33° and
`hitProxy` 0.105/0.098 at 300–450/450+, both better than Pattern's
17.53°/16.19° and 0.104/0.077. **[INFERRED]** At 450+ the required lead
(`mean|req|` = 12.3°) is essentially unpredictable from the past (Pattern
corr 0.165), so Pattern's predicted lead is mostly variance added to a nearly
uninformative signal; a zero-lead aim has error = `|required lead|`, which is
smaller. Consistent with the live record: the live bot's own applied lead
capture was 0.135 at 450+ (job-95), i.e. the live bot was already nearly
zero-lead and hit 9.14 % there; Pattern's offline proxy is 7.7 %.

**[INFERRED / CAVEAT]** The corpus is open-loop: DrussGT's recorded dodge was a
reaction to the LIVE bot's (near-zero-lead) bullets. Replaying Pattern on that
trajectory cannot show what DrussGT would do against Pattern's bullets. This
makes a **live A/B of HeadOn vs Pattern at long range the highest-value cheap
experiment in the campaign** (see §0.6). No offline claim that "HeadOn
beats Pattern" is permitted — only the live A/B decides.

**0.3.5 BitBrain, as shipped, is Pattern [MEASURED].** BitBrain's base is
Pattern and its ADE/SBC corrector changes almost nothing: 450+ `mean|err|`
16.200° vs Pattern 16.193°, `hitProxy` 0.0767 vs 0.0767. TMHorizon likewise
(16.199° / 0.0757). The corrector is currently **adding no measurable aim
information** on this corpus. That is the thing Phase 1 must change.

**0.3.6 Ruler resolution is NOT the limiter [MEASURED].** Aiming at the
integer-tick solve instead of the exact intercept costs only 0.29–1.10° of mean
error (`OracleQuant` column). So the "maybe the oracle only reaches 35 % because
the solve is coarse" worry is dead: the coarse/fine difference is ≈0.3° at long
range, far below the target tolerance (1.93° at 450+). Whatever caps the score,
it is the enemy's unpredictability, not the ruler.

### 0.4 Speed **[MEASURED]**

Full 70-run / 899 607-tick / 3 598 428 tick-bin sweep, 10 arms:
**406.9 s wall**, i.e. `0.1131 ms per tick-bin` over 10 arms,
**≈ 0.045 s per gun per 1000 ticks** (1000 ticks × 4 power bins).
BitBrain is the dominant cost (its ADE pass runs on every `predict` call);
Pattern/TMHorizon cache their per-tick work. A single-arm Pattern-only sweep is
several times cheaper. The binary cache (§0.1) is what makes repeat sweeps
affordable: without it every run re-parses 149 MB of JSONL.

### 0.5 How to reproduce **[MEASURED]**

```
nim c -d:release --nimcache:/tmp/nc_j98 -o:/tmp/bbq_run \
    common_libs/tests/run_prediction_quality.nim
/tmp/bbq_run --corpus /tmp/tfil_ab2/out            # full bar, ~7 min
/tmp/bbq_run --corpus /tmp/tfil_ab2/out --limit 10 # fast subset
/tmp/bbq_run --corpus /tmp/tfil_ab2/out --ruler quant  # integer-tick solve
```
Verbatim full output: `common_libs/tests/prediction_quality_results.txt`.
Determinism: two consecutive full runs are byte-identical except the two
wall-time lines.

**Clean-checkout proof [MEASURED]:** `git archive HEAD | tar -x -C /tmp/bbq_clean`
then, from `/tmp/bbq_clean`,
`nim c -d:release --nimcache:/tmp/nc_j98 -o:bbq_run common_libs/tests/run_prediction_quality.nim`
builds, and `./bbq_run --corpus /tmp/tfil_ab2/out --limit 3` runs and prints the
same tables (separation 13.68× px on the 3-run subset). The committed harness is
self-contained; only the corpus is external.

### 0.6 Designs still to try (seed for later phases)

Ordered by expected value per unit of effort. Phase 0 has already killed one.

| # | design | why it is worth trying | status |
|---|---|---|---|
| D1 | **Live A/B: HeadOn at 450+ vs Pattern** (distance-gated switch, or HeadOn-only control) | Offline says a static gun beats Pattern at long range; the cheapest possible test of the campaign's central premise | **TODO (highest value, live)** |
| D2 | **Pattern variants that raise lead CORRELATION at long range**: longer keys / multi-length keys, per-distance learned pattern tables, different match weighting, k-NN over movement signatures | The lever is corr (0.165 at 450+), not gain; the ruler measures exactly this | TODO (offline-searchable) |
| D3 | **Supervise BitBrain with the ruler's own labels** — per-tick bearing error to the true intercept, trained on N−1 runs, evaluated on a held-out run | BitBrain's corrector currently adds nothing (0.3.5); the ruler gives it a real target. Must hold out runs or it is overfitting | TODO (offline-searchable) |
| D4 | **A causal "predictability" gate**: at each tick estimate whether the future is predictable (e.g. recent pattern-match score, reversal entropy) and fall back to HeadOn/low-variance aim when it is not | Directly attacks the 0.3.4 failure mode without needing a better long-range predictor | TODO |
| D5 | Power policy at long range (already partly done live): lower power = faster bullet = less lead error | Shortens the horizon the predictor must extrapolate; affects hit rate, live-only verdict | TODO (offline proxy only) |
| D6 | Lead-gain sweep 1.0/1.5/2.0/3.0 | **DEAD — measured.** Gain 1.0 wins at every band (§0.3.2) | **KILLED** |

Every D-item must end in a live A/B before any phase verdict.

### 0.7 What would make us quit

> If (a) no causal design raises the 450+ `hitProxy` above the static-gun
> reference (~0.10) on held-out runs by a margin larger than the run-to-run
> spread, **and** (b) the live A/B of the best such design shows no hit-rate or
> damage gain over Pattern with the left-running liveness check satisfied, then
> the campaign stops and we ship the simpler gun. We do not keep tuning an
> offline proxy that has stopped predicting live outcomes.

---

## Phase 1 — *(unclaimed; append below)*

## Phase 2 — *(unclaimed; append below)*