From bfdcdf89196557c4cab33fdaacfeee9f70312879 Mon Sep 17 00:00:00 2001 From: Davide Cappellini Date: Tue, 22 Sep 2026 02:34:05 +0200 Subject: [PATCH] The three unmeasured features, A/B'd - and a methodology correction I had wrong Five arms x 7 runs x 7 rounds (35 real-DrusGT battles, 8 concurrent), one frozen binary from git archive HEAD at 185a32e (includes the shipped Pattern-only rack), env knobs only, server-side event sidecar, exact two-sided permutation tests. arm real % dmg/run survival(rounds won) p vs base base (shipped) 10.61 284.1 16/49 (32.7%) -- nopower (policy off) 7.88 247.1 9/49 (18.4%) 0.0012 oldram (old gates) 10.78 279.4 17/49 (34.7%) 0.6888 ring (tfil_ring) 20.28 267.4 6/49 (12.2%) 0.0006 ringhot (orig heat) 11.73 285.3 15/49 (30.6%) 0.0303 Every round ends with exactly one death (0 timeouts), so survival = round win. 1. POWER POLICY HELPS - KEEP. Turning it off drops real hit rate 10.61 -> 7.88 (p=0.0012), damage/run 284 -> 247, and wins FEWER rounds (16 -> 9). The cap trades per-shot damage for many more shots and a higher per-shot rate; that trade is a clear win. This was shipped on unit tests alone until now. 2. PROACTIVE RAMMING - INDISTINGUISHABLE. oldram vs base p=0.69, damage 279 vs 284, survival 17/49 vs 16/49 (p=1.0), ram contacts 1 vs 2. The lever IS live (6 `opportunity` ON events vs 0 under the old 50px/+30 gates; base reached <40px on 10 ticks vs 0) but converts to essentially no extra collisions and no measurable outcome change. Safe to keep, but it is not earning its keep and reverting it is equally defensible. 3. RING MOVER HURTS THE OBJECTIVE - DO NOT SHIP. And this is the important one. *** METHODOLOGY CORRECTION - I HAD THIS WRONG ALL NIGHT *** The ring arm has the BEST hit rate of anything measured tonight: 20.28% vs 10.61% (+9.67pp, p=0.0006, non-overlapping). Read alone it says "ship it immediately". It is a CONFOUND. The ring halves engagement range (median 460 -> 240px), which halves round length (1542 -> 636 ticks) and shots (4466 -> 1834). So damage/run is FLAT (284 -> 267) while survival/round-win MORE THAN HALVES (16/49 -> 6/49, p=0.0122). It is a GLASS CANNON: same damage dealt, twice as many deaths. The hit-rate gain is a geometric artefact of fighting closer, not an improvement. ** For MOVEMENT arms, hit rate alone INVERTS the verdict. ** A movement change alters range, shots fired and round length simultaneously, so the objective metrics are DAMAGE/RUN and ROUND-WIN RATE (survival) - report all three. For gun/selection arms, where range and round length are held fixed, real hit rate remains the right ground truth. I had been enforcing the hit-rate-only rule without qualification; it is now qualified. HEAT TAMING is the knob that moves the tradeoff: tamed (corridor 5 / wall 10) lets the range weighting pull to ~240px (20.28% / 6 wins); original (20/30) keeps it at ~400px (11.73% / 15 wins). So heat taming buys hit rate at the cost of survival - a knob to keep conservative, and the reason the ring is not the default. Nothing to revert: the shipped movement is already `tfil` and the ring is opt-in. Caveats recorded: only adversary is the real DrussGT jar (the shim hosts only jk.mega.DrussGT), so the ring-vs-winning result needs re-checking elsewhere; and base (10.61%) is consistent with the committed onlyPattern result (9.99%), validating the frozen binary and pipeline. Adds docs/feature_ab_results.md. No source files changed by this job. --- docs/feature_ab_results.md | 69 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+) create mode 100644 docs/feature_ab_results.md diff --git a/docs/feature_ab_results.md b/docs/feature_ab_results.md new file mode 100644 index 0000000..c90ed78 --- /dev/null +++ b/docs/feature_ab_results.md @@ -0,0 +1,69 @@ +# The three unmeasured features, finally A/B'd — and a methodology correction + +Five arms × 7 runs × 7 rounds (35 real-DrusGT bridge battles, 8 concurrent), ONE frozen binary +built from `git archive HEAD` at `185a32e` (includes the shipped Pattern-only rack), env knobs +only, server-side event sidecar ground truth, exact two-sided permutation tests. + +| arm | real % | dmg/run | survival (rounds won) | per-run range | p vs base | +|---|---:|---:|---:|---|---:| +| `base` (shipped) | 10.61 | 284.1 | **16/49** (32.7%) | 8.71–12.18 | — | +| `nopower` (policy off) | 7.88 | 247.1 | 9/49 (18.4%) | 6.51–9.52 | **0.0012** | +| `oldram` (old gates) | 10.78 | 279.4 | 17/49 (34.7%) | 9.67–11.43 | 0.6888 | +| `ring` (`tfil_ring`) | **20.28** | 267.4 | **6/49** (12.2%) | 15.70–23.36 | **0.0006** | +| `ringhot` (ring, original heat) | 11.73 | 285.3 | 15/49 (30.6%) | 10.39–12.45 | 0.0303 | + +Every round ends with exactly one death (0 timeouts), so **survival = round win**. + +## Verdicts + +**1. Power policy — HELPS. Keep.** With it off, real hit rate drops 10.61 → 7.88 (p=0.0012), +damage/run drops 284 → 247, and FEWER rounds are won (16 → 9). The cap trades per-shot damage for +many more shots and a higher per-shot rate, and that trade is a clear win. + +**2. Proactive ramming (200px / +15 energy gates) — INDISTINGUISHABLE.** `oldram` vs `base`: +p=0.69, damage 279 vs 284, survival 17/49 vs 16/49 (p=1.0), ram contacts 1 vs 2. The lever is +provably LIVE (6 `opportunity` ON events vs **0** under the old 50px/+30 gates, and `base` reached +<40px on 10 ticks vs 0) — but it converts to essentially no extra collisions and no measurable +outcome change. Safe to keep; it is not earning its keep, and reverting it is equally defensible. + +**3. Range-weighted ring mover — HURTS the objective. Do NOT ship.** See below; this is the +important one. + +## METHODOLOGY CORRECTION: "real hit rate is the only ground truth" is UNSAFE for movement + +The ring arm has the best hit rate of anything measured all night — **20.28% vs 10.61% (+9.67pp, +p=0.0006, non-overlapping ranges)**. Read alone, that says "ship it immediately". + +It is a confound. The ring halves engagement range (median **460 → 240 px**), which halves round +length (**1542 → 636 ticks**) and shots (**4466 → 1834**). So: + +- **damage/run is FLAT**: 284 → 267 +- **survival / round-win more than halves**: 16/49 → **6/49** (p=0.0122, i.e. base survives + significantly more) +- the bot dies ~2.4× faster + +It is a **glass cannon**: the same damage dealt, twice as many deaths. The hit-rate gain is a +geometric artefact of fighting closer, not an improvement. + +**Rule for movement arms: hit rate alone INVERTS the verdict.** A movement change alters range, +shots fired and round length simultaneously, so the objective metrics are **damage/run and +round-win rate (survival)**. Report all three. (For gun/selection arms, where range and round +length are held fixed, real hit rate remains the right ground truth.) + +## Heat taming is the knob that moves the tradeoff + +Tamed heat (`corridor 5 / wall 10`) is what lets the range weighting pull the bot to ~240px +(20.28% / 6 wins). The original heat (`20/30`) keeps it at ~400px (11.73% / 15 wins, only p=0.0303 +above base). So heat taming buys hit rate at the cost of survival — **a knob to keep conservative**, +and the reason the ring is not the default. + +Nothing to revert: the shipped movement is already `tfil`, and the ring is opt-in. + +## Caveats + +- **Only adversary: the real DrussGT jar** (the shim only hosts `jk.mega.DrussGT`). The ring-vs-winning + result should be re-checked on other bots before any final decision. +- `base` (10.61%) is consistent with the committed `onlyPattern` result (9.99%), validating the + frozen binary and pipeline. +- The observability envs (`TR_POWER_LOG`, `TR_RAM_LOG`, `TR_MOVEMENT_LOG`) were enabled uniformly + across all arms; they gate `echo` only and cannot alter decisions.