SirStone
185a32e9eb
Follow-up to 9cd6e9b, which found the LINEAR base systematically overshoots (mean
radial error -71..-100px, enemy nearer in 63-81% of shots). The question was
whether the gun that actually ships, `Pattern`, overshoots too - because
correcting a systematic bias would be a cheap win.
1. PATTERN DOES NOT OVERSHOOT. Measured over the DrussGT fixtures (n=250,989):
Pattern mean -12.0 px, median -3.2 px, nearer 52.8% / farther 45.0%
Linear mean -87.3 px, median -61.0 px, nearer 83.4% / farther 14.4% (same states)
So the overshoot was a property of the CONSTANT-VELOCITY BASE, not of our
predictions in general. Pattern's pattern-matching does not have it, so there
was nothing to correct. (All-10-fixture pooled: mean -14.0, median -4.2.)
2. THE AVENUE IS STRUCTURALLY DEAD, not merely unprofitable. The live aim is
`aimAngle(self, pred)` and a RADIAL-only offset keeps the BEARING unchanged
(proven exactly by a guard test: bearing is invariant). So a radial offset
cannot change the fired bullet's direction at all. `bmPath` never scores the
aim distance either - and measured, every offset arm is BYTE-IDENTICAL to plain
Pattern on bmPath (33.9%/25.6%). The only real-effect channel is the `shouldFire`
gate via `distPx`, which is indistinguishable from noise.
3. LIVE A/B CONFIRMS: one frozen binary (built from HEAD + only this change),
env-only arms, 7 runs x 7 rounds, 8 concurrent, real DrussGT, server-side hit
rate, exact two-sided permutation test.
control (plain Pattern) 10.61% / 284 dmg-per-run
s0.98 10.89% / 302 (+0.28pp, p=0.62)
s0.95 10.02% (p=0.35)
o-20 10.19% (p=0.46)
No significant winner.
VERDICT: STOP. This line cannot help the shipped configuration, and the reason is
structural rather than statistical - a radial correction is bearing-invariant, so
it is invisible to the actual shot. The bmPoint "win" the radial TM showed was a
metric artefact of that same irrelevance.
Incidental: the control arm (10.61% / 284) independently replicates the shipped
Pattern-only default's A/B numbers (10.36% / 264, 10.78% / 287, 9.99%).
Kept anyway: `TR_PATTERN_RAD_SCALE` / `TR_PATTERN_RAD_OFFSET` default to
(1.0, 0.0) and the default path is byte-identical (proven over 2400 predictions,
plus bearing invariance and unparsable-value fallback - 6 checks). Adds
measure_pattern_radial.nim, sweep_pattern_radial.nim, test_pattern_radial_offset.nim
and pattern_radial_results.md.
Guards: test_gun_harness 39, test_vbullet_metric 11, test_power_selection 3,
test_adaptive_radar 41, test_tfil_ring_weights 24, test_power_policy 26,
test_ram_decision 28, test_rack_membership 48, test_tm_pattern_registration 20,
acceptance_offline_vs_online 12/12 PASS.
246 lines
12 KiB
Markdown
246 lines
12 KiB
Markdown
# Chasing the radial overshoot into the SHIPPED `Pattern` gun
|
||
|
||
Date: 2026-09-22. Author: background worker (executor-heavy).
|
||
Artifacts: `common_libs/guns/pattern_matcher.nim` (radial knob),
|
||
`common_libs/tests/measure_pattern_radial.nim` (measurement),
|
||
`common_libs/tests/test_pattern_radial_offset.nim` (default-parity guard),
|
||
`common_libs/tests/sweep_pattern_radial.nim` (offline sweep).
|
||
Raw outputs: `/tmp/pattern_radial_core.txt`, `/tmp/sp_point.txt`, `/tmp/sp_path.txt`,
|
||
`/tmp/patternrad/analysis2.txt`. Do NOT commit.
|
||
|
||
## The question
|
||
|
||
Commit `9cd6e9b` established that the base LINEAR prediction **systematically
|
||
OVERSHOOTS** range against these range-holding surfers: raw per-tick radial error
|
||
mean −71 to −100 px, enemy NEARER than predicted in 63–81% of fired bullets and
|
||
farther in only 4–14%, consistent across all six captures. A constant short
|
||
offset (aim distance ×0.95 or a fixed −20 px) matched or beat the learned radial
|
||
TM on `bmPoint`. The gun that now ships is **`Pattern`**
|
||
(`guns/pattern_matcher.nim`, rack id 5), not `Linear`. The task: measure Pattern's
|
||
own radial error and, if it overshoots, sweep a constant radial offset offline and
|
||
then A/B it LIVE on real hit rate.
|
||
|
||
## TASK 1 — Pattern's radial error (MEASURED)
|
||
|
||
`measure_pattern_radial.nim` replays each fixture with a FRESH `PatternMatcherGun`
|
||
per round (the offline-range methodology) and, for every fired virtual bullet,
|
||
computes
|
||
|
||
```
|
||
radialErr = |actual enemy pos at the base arrival tick - fire pos|
|
||
- |Pattern's predicted aim point - fire pos|
|
||
```
|
||
|
||
with the arrival tick mirroring the resolver exactly
|
||
(`arrOff = max(0, ceil(aimDist/speed) - 1)`). Negative = enemy NEARER =
|
||
Pattern OVERSHOT. The identical states are also run through `forecastLinear` for
|
||
reference. Pooled over the six captures of the `9cd6e9b` table
|
||
(n = 250 989 Pattern bullets):
|
||
|
||
| gun | mean px | median px | meanAbs px | % nearer (<0) | % farther (>0) | % nearer (<−18) | % farther (>+18) | p10 | p90 |
|
||
|---|---|---|---|---|---|---|---|---|---|
|
||
| **Pattern** | **−12.0** | **−3.2** | 47.9 | **52.8** | **45.0** | 38.6 | 30.3 | −90.4 | 60.4 |
|
||
| Linear base | −87.3 | −61.0 | 95.4 | 83.4 | 14.4 | 73.4 | 7.9 | −225.6 | 11.2 |
|
||
|
||
Per capture (Pattern mean / median / % nearer / % farther):
|
||
`crazy` −31.4 / −14.4 / 58.2 / 30.3; `spinbot` −39.0 / −25.0 / 64.7 / 31.3;
|
||
`drussgt` −1.5 / +1.0 / 46.7 / 50.9; `tr_crazy` −12.2 / −6.1 / 54.3 / 45.7;
|
||
`tr_spinbot` −9.0 / −1.6 / 52.0 / 48.0; `tr_modularbot` −1.6 / +1.0 / 49.0 / 51.0.
|
||
|
||
Over the FULL `tools/fixtures/*drussgt*.jsonl` set (all ten files, n = 344 123) the
|
||
picture is the same: Pattern mean −14.0 / median −4.2, near/far **53.5% / 44.1%**
|
||
(Linear −87.4 / −60.8, 82.5% / 15.1%). The three extra captures show Pattern's
|
||
largest residual (`ramfire` −49.7 / −33.5, `tr_corners` −42.3 / −36.5,
|
||
`corners` −23.7 / −14.4) — still a fraction of the base's bias on the same files.
|
||
Raw: `/tmp/pattern_radial_all.txt`.
|
||
|
||
Error distribution (Pattern, pooled, 20 px bins):
|
||
`[−20,20)` 33.7%, `[20,60)` 18.9%, `[60,100)` 7.0%, `[−60,−20)` 19.5%,
|
||
`[−100,−60)` 9.4%, `[−150,−100)` 4.7%, `[100,150)` 2.4%, `<−150` 3.6%,
|
||
`>+150` 0.6%.
|
||
|
||
**VERDICT (MEASURED): Pattern does NOT reproduce the base's systematic
|
||
overshoot.** Its pooled median error is **−3.2 px** (base: −61 px) and the
|
||
nearer/farther split is **52.8% / 45.0%** (base: 83.4% / 14.4%) — nearly
|
||
symmetric. The base's overshoot is a property of the *constant-velocity
|
||
extrapolation*: it lets range grow geometrically while a surfer holds it. Pattern
|
||
replays matched movement, so it already reproduces the deceleration/turn and the
|
||
bias is largely gone. A weak, **adversary-specific** residual remains on
|
||
`crazy`/`spinbot` (median −14/−25 px), but it is nowhere near the base's
|
||
systematic −61 px, and on `drussgt`/`tr_modularbot` Pattern is slightly net-LONG.
|
||
|
||
## TASK 2 — the env knob (default path byte-identical)
|
||
|
||
`pattern_matcher.nim` gains two runtime knobs, read lazily on first prediction:
|
||
|
||
* `TR_PATTERN_RAD_SCALE` (default **1.0**) — multiplier on the predicted aim distance.
|
||
* `TR_PATTERN_RAD_OFFSET` (default **0.0**) — px added to the predicted aim distance.
|
||
|
||
The BEARING is untouched; only the distance changes:
|
||
`aim = self + unit(pred−self) * max(0, dist*scale + offset)`. An explicit setter
|
||
`setRadialCorrection(scale, offsetPx)` (used by the offline sweep) writes the same
|
||
fields, so the measured and live paths are identical.
|
||
|
||
**Default parity (MEASURED, `test_pattern_radial_offset.nim`):** the
|
||
`(1.0, 0.0)` case short-circuits to the raw aim point, and an unconfigured gun
|
||
(env unset) is **bit-for-bit identical** to one explicitly set to `(1.0, 0.0)`
|
||
over 2400 predictions on a real DrussGT fixture. The sweep independently
|
||
confirms it: `Pattern` and `P_s1.00` are identical on both metrics. Six assertions
|
||
pass, including that scale/offset change the distance but leave the bearing
|
||
exactly unchanged, and that unparsable env values fall back to `(1.0, 0.0)`.
|
||
|
||
## TASK 3a — offline sweep (MEASURED)
|
||
|
||
`Pattern` is deterministic, so each fixture is replayed once per arm (fresh gun).
|
||
|
||
### bmPoint
|
||
|
||
| arm | early | overall |
|
||
|---|---|---|
|
||
| Linear | 7.2% | 4.7% |
|
||
| **Pattern** | **11.0%** | **9.1%** |
|
||
| `P_s1.00` | 11.0% | 9.1% *(identical to Pattern)* |
|
||
| `P_s0.98` | 11.7% | 9.3% *(+0.2 pp; per-fixture 3/3 tie, p=1.0)* |
|
||
| `P_s0.95` | 11.8% | 7.5% *(worse overall)* |
|
||
| `P_s0.90` | 6.8% | 3.6% |
|
||
| `P_s0.85` | 5.0% | 2.3% |
|
||
| `P_o−10` | 12.0% | 9.1% *(tie)* |
|
||
| `P_o−20` | 10.7% | 7.7% |
|
||
| `P_o−30` | 6.8% | 4.7% |
|
||
| `P_o−40` | 4.8% | 3.2% |
|
||
| `P_o+30` | 3.1% | 3.0% |
|
||
|
||
**No offset improves `bmPoint`.** The best arm is scale 0.98 at +0.2 pp overall
|
||
with a per-fixture 3/3 tie (p=1.0); anything past ~−10 px is a significant loss.
|
||
This is the opposite of the LINEAR base, where scale 0.95 gave a real `bmPoint`
|
||
win: Pattern's pattern matcher has already spent the radial headroom. (The
|
||
`+30` control collapsing to 3.0% confirms the measured direction, but the
|
||
direction is already right at baseline.)
|
||
|
||
### bmPath (the SHIPPED metric) — LOUD DEAD END
|
||
|
||
| arm | early | overall |
|
||
|---|---|---|
|
||
| Linear | 34.0% | 24.3% |
|
||
| **Pattern** | **33.9%** | **25.6%** |
|
||
| every scale arm 1.00…0.85 | 33.9% | 25.6% |
|
||
| every fixed arm −10…−30, +30 | 33.9% | 25.6% |
|
||
| `P_o−40` | 33.9% | 25.6% *(8 early hits fewer — a clamp edge)* |
|
||
|
||
**Under `bmPath` the radial offset is an EXACT structural no-op: every arm is
|
||
byte-identical to plain Pattern.** The bearing (and therefore the ray) is
|
||
unchanged, and `bmPath` never scores the aim distance. So **this whole line of
|
||
work CANNOT help the shipped configuration** — exactly the dead end the LINEAR
|
||
base hit, and now proven for the shipped gun. Do not pursue radial offsets on
|
||
`bmPath`.
|
||
|
||
## TASK 3b — the LIVE A/B (the decider)
|
||
|
||
ONE frozen binary (`/tmp/ModularBot_patternrad`, built from `HEAD` + only the
|
||
`pattern_matcher.nim` change, so the concurrently-edited `ModularBot.nim` /
|
||
`virtual_bullets.nim` could not contaminate it). Every arm is the SAME binary
|
||
with env only: `TR_RACK_*` isolates `Pattern`, plus the knob. 7 runs × 7 rounds
|
||
per arm, 8 concurrent battles, server-side real hit rate from the events sidecar,
|
||
exact two-sided permutation test on per-run rates.
|
||
|
||
| arm | runs | shots | hits | real % | dmg/run | per-run range | Δ vs control | p |
|
||
|---|---|---|---|---|---|---|---|---|
|
||
| `onlyPattern` (control) | 7 | 4563 | 484 | **10.61** | 284 | 9.32–11.75 | — | — |
|
||
| `p_s098` (scale 0.98) | 7 | 4718 | 514 | 10.89 | 302 | 9.64–12.46 | +0.28 pp | **0.62** |
|
||
| `p_s095` (scale 0.95) | 7 | 4371 | 438 | 10.02 | 259 | 7.97–11.83 | −0.59 pp | **0.35** |
|
||
| `p_o20` (offset −20 px) | 7 | 3306 | 337 | 10.19 | 201 | 3.17–11.76 | −0.42 pp | **0.46** |
|
||
|
||
Liveness (MEASURED): every arm selected `Pattern` on 100% of ticks; no other gun
|
||
was selected. So the arms differ only by the knob.
|
||
|
||
**VERDICT (MEASURED): no offset significantly improves Pattern's real hit rate.**
|
||
The best arm (scale 0.98) is +0.28 pp with p = 0.62; the ranges overlap
|
||
completely. There is no winner to declare.
|
||
|
||
**Why this is expected (INFERRED, and now structurally supported):** the live
|
||
shot direction is `aimAngle(self, pred)`. A radial-only offset leaves the
|
||
bearing unchanged (proven exactly by the guard test), so it cannot change the
|
||
real bullet's trajectory. The only real-effect channel is the
|
||
range-aware fire gate (`shouldFire(..., distPx)` with
|
||
`distPx = |pred − self|`): shrinking the distance loosens the gate slightly.
|
||
That channel is real but tiny, and the A/B cannot distinguish it from noise.
|
||
|
||
## Guard suites (MEASURED)
|
||
|
||
| suite | checks | result |
|
||
|---|---|---|
|
||
| `test_gun_harness` | 39 | pass |
|
||
| `test_vbullet_metric` | 11 | pass |
|
||
| `test_power_selection` | 3 | pass |
|
||
| `test_adaptive_radar` | 41 | pass |
|
||
| `test_tfil_ring_weights` | 24 | pass |
|
||
| `test_power_policy` | 26 | pass |
|
||
| `test_ram_decision` | 28 | pass |
|
||
| `test_rack_membership` | **48** (was 38; the concurrent rack change added 10) | pass |
|
||
| `test_tm_pattern_registration` | 20 | pass |
|
||
| `test_pattern_radial_offset` | 6 | pass (new) |
|
||
|
||
`test_rack_membership` now reports 48, not 38 — the committed rack-default change
|
||
(`31c7c01`) added assertions. See the acceptance note below.
|
||
|
||
## DIRECT VERDICT
|
||
|
||
1. **Pattern does NOT overshoot the way the base did.** Pooled median radial error
|
||
−3.2 px and a near/far split of 52.8%/45.0%, versus the base's −61 px and
|
||
83.4%/14.4%. Pattern's pattern matcher already absorbs the range-holding error
|
||
the base mispredicts.
|
||
2. **The offset knob is safe**: env-tunable, defaults `(1.0, 0.0)`, and the
|
||
default path is proven bit-for-bit identical to the pre-knob gun.
|
||
3. **bmPath (shipped): exact structural no-op.** Every offset arm is byte-identical
|
||
to plain Pattern. The radial avenue cannot help the shipped configuration.
|
||
4. **bmPoint: no headroom left.** No offset improves it; the best is +0.2 pp at
|
||
p = 1.0.
|
||
5. **LIVE: no significant effect.** Best arm +0.28 pp, p = 0.62. The radial offset
|
||
does not change the real aim direction, so this is expected.
|
||
|
||
**STOP. Do not ship a radial offset on Pattern.** The `9cd6e9b` overshoot finding
|
||
was real for the LINEAR base and has no exploitable analog in the shipped gun.
|
||
|
||
## MEASURED vs INFERRED
|
||
|
||
* MEASURED: the Pattern radial distribution (mean/median/abs/percentiles and the
|
||
nearer/farther split), the Linear reference on the identical states, the full
|
||
bmPoint and bmPath offline tables, the default-parity byte-identity and the
|
||
bearing invariance, the live per-run rates and the exact permutation p-values,
|
||
and every guard count.
|
||
* MEASURED: under `bmPath` every offset arm is byte-identical to plain Pattern.
|
||
* INFERRED: that the live effect can only flow through the fire gate
|
||
(`distPx`), because the bearing is provably unchanged; the A/B cannot separate
|
||
that tiny channel from noise.
|
||
* INFERRED (not measured): whether a per-adversary offset would help; the
|
||
per-fixture optima in the bmPoint table are in-sample and the live test used a
|
||
single global constant.
|
||
|
||
## Caveats
|
||
|
||
* The `p_o20` arm recorded 29 rounds / 3306 shots versus 42–43 rounds / ~4560
|
||
shots for the control (some fights ended earlier). This does not change the
|
||
verdict (it is not the best arm and is not significant), but the arm has less
|
||
data than the others.
|
||
* `acceptance_offline_vs_online` was launched while another worker was
|
||
concurrently editing `ModularBot.nim`, `virtual_bullets.nim` and
|
||
`acceptance_offline_vs_online.nim`; its result is reported separately.
|
||
|
||
## How to reproduce
|
||
|
||
```bash
|
||
nim c -r -d:release --path:common_libs \
|
||
common_libs/tests/measure_pattern_radial.nim --set=core
|
||
nim c -r -d:release --path:common_libs \
|
||
common_libs/tests/test_pattern_radial_offset.nim
|
||
nim c -r -d:release --path:common_libs \
|
||
common_libs/tests/sweep_pattern_radial.nim --metric=point
|
||
nim c -r -d:release --path:common_libs \
|
||
common_libs/tests/sweep_pattern_radial.nim --metric=path
|
||
|
||
# one frozen binary; 4 arms x 7 runs x 7 rounds, 8 concurrent
|
||
cd /tmp/patternrad && cat jobs.txt | xargs -P 8 -n 2 ./run_one.sh
|
||
WHICHGUN_OUT=/tmp/patternrad python3 tools/ab/which_gun_analyze.py \
|
||
onlyPattern p_s098 p_s095 p_o20
|
||
```
|