diff --git a/docs/feature_ab_results.md b/docs/feature_ab_results.md new file mode 100644 index 0000000..c90ed78 --- /dev/null +++ b/docs/feature_ab_results.md @@ -0,0 +1,69 @@ +# The three unmeasured features, finally A/B'd — and a methodology correction + +Five arms × 7 runs × 7 rounds (35 real-DrusGT bridge battles, 8 concurrent), ONE frozen binary +built from `git archive HEAD` at `185a32e` (includes the shipped Pattern-only rack), env knobs +only, server-side event sidecar ground truth, exact two-sided permutation tests. + +| arm | real % | dmg/run | survival (rounds won) | per-run range | p vs base | +|---|---:|---:|---:|---|---:| +| `base` (shipped) | 10.61 | 284.1 | **16/49** (32.7%) | 8.71–12.18 | — | +| `nopower` (policy off) | 7.88 | 247.1 | 9/49 (18.4%) | 6.51–9.52 | **0.0012** | +| `oldram` (old gates) | 10.78 | 279.4 | 17/49 (34.7%) | 9.67–11.43 | 0.6888 | +| `ring` (`tfil_ring`) | **20.28** | 267.4 | **6/49** (12.2%) | 15.70–23.36 | **0.0006** | +| `ringhot` (ring, original heat) | 11.73 | 285.3 | 15/49 (30.6%) | 10.39–12.45 | 0.0303 | + +Every round ends with exactly one death (0 timeouts), so **survival = round win**. + +## Verdicts + +**1. Power policy — HELPS. Keep.** With it off, real hit rate drops 10.61 → 7.88 (p=0.0012), +damage/run drops 284 → 247, and FEWER rounds are won (16 → 9). The cap trades per-shot damage for +many more shots and a higher per-shot rate, and that trade is a clear win. + +**2. Proactive ramming (200px / +15 energy gates) — INDISTINGUISHABLE.** `oldram` vs `base`: +p=0.69, damage 279 vs 284, survival 17/49 vs 16/49 (p=1.0), ram contacts 1 vs 2. The lever is +provably LIVE (6 `opportunity` ON events vs **0** under the old 50px/+30 gates, and `base` reached +<40px on 10 ticks vs 0) — but it converts to essentially no extra collisions and no measurable +outcome change. Safe to keep; it is not earning its keep, and reverting it is equally defensible. + +**3. Range-weighted ring mover — HURTS the objective. Do NOT ship.** See below; this is the +important one. + +## METHODOLOGY CORRECTION: "real hit rate is the only ground truth" is UNSAFE for movement + +The ring arm has the best hit rate of anything measured all night — **20.28% vs 10.61% (+9.67pp, +p=0.0006, non-overlapping ranges)**. Read alone, that says "ship it immediately". + +It is a confound. The ring halves engagement range (median **460 → 240 px**), which halves round +length (**1542 → 636 ticks**) and shots (**4466 → 1834**). So: + +- **damage/run is FLAT**: 284 → 267 +- **survival / round-win more than halves**: 16/49 → **6/49** (p=0.0122, i.e. base survives + significantly more) +- the bot dies ~2.4× faster + +It is a **glass cannon**: the same damage dealt, twice as many deaths. The hit-rate gain is a +geometric artefact of fighting closer, not an improvement. + +**Rule for movement arms: hit rate alone INVERTS the verdict.** A movement change alters range, +shots fired and round length simultaneously, so the objective metrics are **damage/run and +round-win rate (survival)**. Report all three. (For gun/selection arms, where range and round +length are held fixed, real hit rate remains the right ground truth.) + +## Heat taming is the knob that moves the tradeoff + +Tamed heat (`corridor 5 / wall 10`) is what lets the range weighting pull the bot to ~240px +(20.28% / 6 wins). The original heat (`20/30`) keeps it at ~400px (11.73% / 15 wins, only p=0.0303 +above base). So heat taming buys hit rate at the cost of survival — **a knob to keep conservative**, +and the reason the ring is not the default. + +Nothing to revert: the shipped movement is already `tfil`, and the ring is opt-in. + +## Caveats + +- **Only adversary: the real DrussGT jar** (the shim only hosts `jk.mega.DrussGT`). The ring-vs-winning + result should be re-checked on other bots before any final decision. +- `base` (10.61%) is consistent with the committed `onlyPattern` result (9.99%), validating the + frozen binary and pipeline. +- The observability envs (`TR_POWER_LOG`, `TR_RAM_LOG`, `TR_MOVEMENT_LOG`) were enabled uniformly + across all arms; they gate `echo` only and cannot alter decisions.