SirStone
bfdcdf8919
Five arms x 7 runs x 7 rounds (35 real-DrusGT battles, 8 concurrent), one frozen
binary from git archive HEAD at 185a32e (includes the shipped Pattern-only rack),
env knobs only, server-side event sidecar, exact two-sided permutation tests.
arm real % dmg/run survival(rounds won) p vs base
base (shipped) 10.61 284.1 16/49 (32.7%) --
nopower (policy off) 7.88 247.1 9/49 (18.4%) 0.0012
oldram (old gates) 10.78 279.4 17/49 (34.7%) 0.6888
ring (tfil_ring) 20.28 267.4 6/49 (12.2%) 0.0006
ringhot (orig heat) 11.73 285.3 15/49 (30.6%) 0.0303
Every round ends with exactly one death (0 timeouts), so survival = round win.
1. POWER POLICY HELPS - KEEP. Turning it off drops real hit rate 10.61 -> 7.88
(p=0.0012), damage/run 284 -> 247, and wins FEWER rounds (16 -> 9). The cap
trades per-shot damage for many more shots and a higher per-shot rate; that
trade is a clear win. This was shipped on unit tests alone until now.
2. PROACTIVE RAMMING - INDISTINGUISHABLE. oldram vs base p=0.69, damage 279 vs
284, survival 17/49 vs 16/49 (p=1.0), ram contacts 1 vs 2. The lever IS live
(6 `opportunity` ON events vs 0 under the old 50px/+30 gates; base reached
<40px on 10 ticks vs 0) but converts to essentially no extra collisions and no
measurable outcome change. Safe to keep, but it is not earning its keep and
reverting it is equally defensible.
3. RING MOVER HURTS THE OBJECTIVE - DO NOT SHIP. And this is the important one.
*** METHODOLOGY CORRECTION - I HAD THIS WRONG ALL NIGHT ***
The ring arm has the BEST hit rate of anything measured tonight: 20.28% vs 10.61%
(+9.67pp, p=0.0006, non-overlapping). Read alone it says "ship it immediately".
It is a CONFOUND. The ring halves engagement range (median 460 -> 240px), which
halves round length (1542 -> 636 ticks) and shots (4466 -> 1834). So damage/run is
FLAT (284 -> 267) while survival/round-win MORE THAN HALVES (16/49 -> 6/49,
p=0.0122). It is a GLASS CANNON: same damage dealt, twice as many deaths. The
hit-rate gain is a geometric artefact of fighting closer, not an improvement.
** For MOVEMENT arms, hit rate alone INVERTS the verdict. ** A movement change
alters range, shots fired and round length simultaneously, so the objective
metrics are DAMAGE/RUN and ROUND-WIN RATE (survival) - report all three. For
gun/selection arms, where range and round length are held fixed, real hit rate
remains the right ground truth. I had been enforcing the hit-rate-only rule
without qualification; it is now qualified.
HEAT TAMING is the knob that moves the tradeoff: tamed (corridor 5 / wall 10)
lets the range weighting pull to ~240px (20.28% / 6 wins); original (20/30) keeps
it at ~400px (11.73% / 15 wins). So heat taming buys hit rate at the cost of
survival - a knob to keep conservative, and the reason the ring is not the default.
Nothing to revert: the shipped movement is already `tfil` and the ring is opt-in.
Caveats recorded: only adversary is the real DrussGT jar (the shim hosts only
jk.mega.DrussGT), so the ring-vs-winning result needs re-checking elsewhere; and
base (10.61%) is consistent with the committed onlyPattern result (9.99%),
validating the frozen binary and pipeline.
Adds docs/feature_ab_results.md. No source files changed by this job.
70 lines
3.9 KiB
Markdown
70 lines
3.9 KiB
Markdown
# The three unmeasured features, finally A/B'd — and a methodology correction
|
||
|
||
Five arms × 7 runs × 7 rounds (35 real-DrusGT bridge battles, 8 concurrent), ONE frozen binary
|
||
built from `git archive HEAD` at `185a32e` (includes the shipped Pattern-only rack), env knobs
|
||
only, server-side event sidecar ground truth, exact two-sided permutation tests.
|
||
|
||
| arm | real % | dmg/run | survival (rounds won) | per-run range | p vs base |
|
||
|---|---:|---:|---:|---|---:|
|
||
| `base` (shipped) | 10.61 | 284.1 | **16/49** (32.7%) | 8.71–12.18 | — |
|
||
| `nopower` (policy off) | 7.88 | 247.1 | 9/49 (18.4%) | 6.51–9.52 | **0.0012** |
|
||
| `oldram` (old gates) | 10.78 | 279.4 | 17/49 (34.7%) | 9.67–11.43 | 0.6888 |
|
||
| `ring` (`tfil_ring`) | **20.28** | 267.4 | **6/49** (12.2%) | 15.70–23.36 | **0.0006** |
|
||
| `ringhot` (ring, original heat) | 11.73 | 285.3 | 15/49 (30.6%) | 10.39–12.45 | 0.0303 |
|
||
|
||
Every round ends with exactly one death (0 timeouts), so **survival = round win**.
|
||
|
||
## Verdicts
|
||
|
||
**1. Power policy — HELPS. Keep.** With it off, real hit rate drops 10.61 → 7.88 (p=0.0012),
|
||
damage/run drops 284 → 247, and FEWER rounds are won (16 → 9). The cap trades per-shot damage for
|
||
many more shots and a higher per-shot rate, and that trade is a clear win.
|
||
|
||
**2. Proactive ramming (200px / +15 energy gates) — INDISTINGUISHABLE.** `oldram` vs `base`:
|
||
p=0.69, damage 279 vs 284, survival 17/49 vs 16/49 (p=1.0), ram contacts 1 vs 2. The lever is
|
||
provably LIVE (6 `opportunity` ON events vs **0** under the old 50px/+30 gates, and `base` reached
|
||
<40px on 10 ticks vs 0) — but it converts to essentially no extra collisions and no measurable
|
||
outcome change. Safe to keep; it is not earning its keep, and reverting it is equally defensible.
|
||
|
||
**3. Range-weighted ring mover — HURTS the objective. Do NOT ship.** See below; this is the
|
||
important one.
|
||
|
||
## METHODOLOGY CORRECTION: "real hit rate is the only ground truth" is UNSAFE for movement
|
||
|
||
The ring arm has the best hit rate of anything measured all night — **20.28% vs 10.61% (+9.67pp,
|
||
p=0.0006, non-overlapping ranges)**. Read alone, that says "ship it immediately".
|
||
|
||
It is a confound. The ring halves engagement range (median **460 → 240 px**), which halves round
|
||
length (**1542 → 636 ticks**) and shots (**4466 → 1834**). So:
|
||
|
||
- **damage/run is FLAT**: 284 → 267
|
||
- **survival / round-win more than halves**: 16/49 → **6/49** (p=0.0122, i.e. base survives
|
||
significantly more)
|
||
- the bot dies ~2.4× faster
|
||
|
||
It is a **glass cannon**: the same damage dealt, twice as many deaths. The hit-rate gain is a
|
||
geometric artefact of fighting closer, not an improvement.
|
||
|
||
**Rule for movement arms: hit rate alone INVERTS the verdict.** A movement change alters range,
|
||
shots fired and round length simultaneously, so the objective metrics are **damage/run and
|
||
round-win rate (survival)**. Report all three. (For gun/selection arms, where range and round
|
||
length are held fixed, real hit rate remains the right ground truth.)
|
||
|
||
## Heat taming is the knob that moves the tradeoff
|
||
|
||
Tamed heat (`corridor 5 / wall 10`) is what lets the range weighting pull the bot to ~240px
|
||
(20.28% / 6 wins). The original heat (`20/30`) keeps it at ~400px (11.73% / 15 wins, only p=0.0303
|
||
above base). So heat taming buys hit rate at the cost of survival — **a knob to keep conservative**,
|
||
and the reason the ring is not the default.
|
||
|
||
Nothing to revert: the shipped movement is already `tfil`, and the ring is opt-in.
|
||
|
||
## Caveats
|
||
|
||
- **Only adversary: the real DrussGT jar** (the shim only hosts `jk.mega.DrussGT`). The ring-vs-winning
|
||
result should be re-checked on other bots before any final decision.
|
||
- `base` (10.61%) is consistent with the committed `onlyPattern` result (9.99%), validating the
|
||
frozen binary and pipeline.
|
||
- The observability envs (`TR_POWER_LOG`, `TR_RAM_LOG`, `TR_MOVEMENT_LOG`) were enabled uniformly
|
||
across all arms; they gate `echo` only and cannot alter decisions.
|