SirStone
9cd6e9b8ce
shipped metric The radial TM beats Linear on bmPoint, but its head never beat the majority baseline after the label bias was fixed - suggesting the win is a constant lean rather than learning. So: sweep a stateless constant short-range offset (new `common_libs/guns/radial_offset.nim`, no learning at all) against the learned TM. VERDICT (measured, offline range, seeds=3, 18 paired runs, 231 TM rounds): 1. **bmPoint - REPLACE the TM with a constant.** `RO_s0.95` (aim distance x0.95) TIES it early (9/9, p=1.0) and BEATS it overall (15/3, p=0.0075; 7.47% vs 6.89% per-run mean). A fixed -20px does the same. The head never beats its majority baseline (56.2% vs 57.2%). 2. **bmPath (the SHIPPED metric) - the radial avenue is a DEAD END.** TMRadial is a systematic LOSS there (2/16, p=0.0013); every constant is within +-0.2pp; the only real bmPath effect is the BotRadius clamp. So the radial shift cannot help the shipped configuration. 3. The per-adversary optimum DOES vary (fixed -10 for crazy, -30 for tr_crazy, scale 0.95 for three others) - but ONE GLOBAL CONSTANT still beats the adaptively-trained head, so the "fragility justifies learning" argument FAILS. THE REAL FINDING UNDERNEATH, and it generalises beyond this gun: the base linear prediction systematically OVERSHOOTS. Measured raw per-tick base radial error has mean -71 to -100 px; the enemy is NEARER than the prediction in 63-81% of shots and farther in only 4-14%, CONSISTENT ACROSS ALL SIX CAPTURES. Radial label histogram [415166,126461,120325,44694,19351] = 57.2% majority class, mean label -82.3 px, mean applied shift -37.6 px. So the net-short bias is a GENUINE property of these range-holders against a constant-velocity extrapolation (they decelerate and turn, so the true position is closer than the straight-line guess) - NOT a fixture artefact. That is worth chasing for the guns that actually ship. Caveat: bmPoint is not the shipped metric (bmPath won the real-hit-rate A/B for SELECTION), so a bmPoint win is not yet evidence of a real win. That needs a live test - and the natural target is Pattern, which is now the default and best gun. Adds radial_offset.nim + sweep_radial_offset.nim; tm_pattern.nim gains additive instrumentation only (radial label mean and applied-shift mean; no behaviour change, and test_tm_pattern_registration still passes all 20 checks).
593 lines
30 KiB
Markdown
593 lines
30 KiB
Markdown
# TM pattern gun — discrete-target sweep results
|
||
|
||
Date: 2026-09-21. Author: background worker (executor-heavy).
|
||
Artifacts implementing this: `common_libs/guns/tm_pattern.nim`,
|
||
`common_libs/tests/sweep_tm_pattern.nim`. Do not commit.
|
||
|
||
## What was built
|
||
|
||
`tm_pattern.nim` is a NEW gun (the old `guns/tsetlin.nim` is untouched). It
|
||
attacks both the REPRESENTATION and the TARGET as the brief asked:
|
||
|
||
* **Base**: `forecastLinear` (the exact self-consistent forecast `LinearGun`
|
||
uses). GF class 0 (centre) reproduces the Linear gun byte-for-byte, so any
|
||
measured difference is attributable to the TM.
|
||
* **Target**: a discrete multi-class GUESS-FACTOR BUCKET — which lateral escape
|
||
sector (in max-escape-angle units) the enemy occupied at the tick the bullet
|
||
would have reached the BASE fire distance. 5 or 9 classes.
|
||
* **Label**: read from a per-tick ring of our own recorded enemy positions at the
|
||
base arrival tick, NOT from `FeedbackEvent.actualXY`. Under the shipped
|
||
`bmPath` metric `actualXY` is the closest-approach point on the gun's OWN aim
|
||
ray, which biases the label toward the gun's own last output; the ring gives a
|
||
clean, metric-independent label.
|
||
* **Features**: 40 hand-built binary/bucketed motion features (lateral-velocity
|
||
sign over 3 ticks, turn-rate sign over 3 ticks, time since reversal, lateral
|
||
magnitude, speed/distance/flight-time bands, four per-wall proximity bits,
|
||
radial-fraction band, energy band, heading relative to LOS, approach sign).
|
||
* **TM core**: compact self-contained Granmo Table 2/3 with the corrected
|
||
feedback rules and Eq. 6 empty-clause bootstrap (same corrected core as
|
||
tsetlin.nim / tm_selector.nim, re-derived at 40-bit width).
|
||
* **Per-enemy / freshness**: a fresh net per gun instance; the net and history
|
||
reset if the target id changes. Each offline round is replayed with a fresh
|
||
instance (cold every battle, overfit within the battle).
|
||
|
||
Config overrides used in the final run: `-d:TM_CONF_MARGIN_DEF=0.25
|
||
-d:TM_SHRINK_DEF=0.5` (confidence gate + shrink). Defaults are 0.0 / 1.0
|
||
(= raw argmax). Compile-time knobs: `TM_CLASSES`, `TM_NCLAUSES`, `TM_NSTATES`,
|
||
`TM_S_DEF`, `TM_MIN_OBS`, `TM_CONF_MARGIN_DEF`, `TM_SHRINK_DEF`, `TM_GF_MODE`
|
||
(hard|soft), `TM_SOFT_BETA_DEF`.
|
||
|
||
## How to reproduce
|
||
|
||
```
|
||
nim c --path:common_libs -d:release \
|
||
-d:TM_CONF_MARGIN_DEF=0.25 -d:TM_SHRINK_DEF=0.5 \
|
||
-o:/tmp/sweep_tm_pattern common_libs/tests/sweep_tm_pattern.nim
|
||
/tmp/sweep_tm_pattern --set=real --seeds=3 --metric=path \
|
||
--variants=linear,tsetlin,tmpat,tmpat_shuf
|
||
/tmp/sweep_tm_pattern --set=real --seeds=3 --metric=point \
|
||
--variants=linear,tsetlin,tmpat,tmpat_shuf
|
||
```
|
||
|
||
Raw outputs: `/tmp/final_path_s3.txt`, `/tmp/final_point_s3.txt`,
|
||
`/tmp/final_ungated_path_s3.txt`, `/tmp/syn_*`.
|
||
|
||
## Metric
|
||
|
||
EARLY = resolutions in the first 100 ticks of each round (a cold TM every
|
||
round). OVERALL = whole fixture. Pooled over all rounds / fixtures / seeds.
|
||
`TMPatternShuf` = identical gun/encoding/cadence but the training label is a
|
||
uniform-random class (the mandatory shuffled-feedback control). Per-run = one
|
||
fixture × one seed (Linear is deterministic and replicated across seeds for
|
||
pairing). Significance = exact two-sided paired sign test, 18 pairs.
|
||
|
||
## The core result — the discrete target IS learnable, but does not beat the base
|
||
|
||
Online classification accuracy of the GF bucket (warm predictions only,
|
||
seeds=1, n ≈ 1.26 M for each arm):
|
||
|
||
| arm | correct/total | accuracy |
|
||
|---|---|---|
|
||
| TMPattern (real labels) | 578722/1258488 | **46.0%** |
|
||
| TMPatternShuf (random labels) | 246733/1231116 | **20.0%** (chance) |
|
||
|
||
So the Tsetlin Machine genuinely learns the discrete target (2.3× chance). The
|
||
representation mismatch was real and is fixed. The problem is that the target
|
||
is not aligned with what wins the metric.
|
||
|
||
### Real DrussGT fixtures, bmPath (shipped), seeds=3
|
||
|
||
| variant | early | overall |
|
||
|---|---|---|
|
||
| Linear | 34.0% (6358/18715) | 24.3% (58297/239943) |
|
||
| Tsetlin (default) | 22.3% (12344/55463) | 20.3% (145828/719205) |
|
||
| **TMPattern (gated)** | **27.9% (15514/55535)** | **22.0% (158658/719681)** |
|
||
| TMPatternShuf | 28.7% (16049/55969) | 19.4% (139639/719790) |
|
||
|
||
Paired sign tests (18 runs; ranges overlap, so the paired test is the test):
|
||
|
||
* Linear > TMPattern: early 15/18 p=0.0075; overall 15/18 p=0.0075. **Significantly
|
||
worse than Linear.**
|
||
* TMPattern > TMPatternShuf: early 10/8 p=0.81 (tie); overall 17/1 p=0.0001.
|
||
**Learning is real but shows up mainly in the whole-round aggregate, not early.**
|
||
* TMPattern > Tsetlin: early 17/1 p=0.0001; overall 12/6 p=0.24. **Beats the
|
||
default TM gun early, ties overall.**
|
||
|
||
Per-run distributions (mean [min,max], 18 runs):
|
||
Linear early 35.52 [26.03,50.55] / overall 26.86 [9.79,43.36];
|
||
Tsetlin 25.76 [19.79,47.68] / 21.99 [9.64,31.34];
|
||
TMPattern 31.18 [20.17,55.11] / 24.45 [11.24,37.78];
|
||
Shuf 31.42 [20.72,53.49] / 21.07 [9.55,32.65].
|
||
|
||
### Raw ungated hard argmax (margin 0.0, shrink 1.0), bmPath, seeds=3
|
||
|
||
| variant | early | overall |
|
||
|---|---|---|
|
||
| Linear | 34.0% | 24.3% |
|
||
| Tsetlin | 22.3% | 20.3% |
|
||
| TMPattern | 21.2% (11783/55664) | 18.6% (133465/719432) |
|
||
| TMPatternShuf | 15.2% (8710/57169) | 8.3% (59903/720583) |
|
||
|
||
TMPattern > Shuf 18/18 p<0.0001 on BOTH early and overall; TMPattern < Linear
|
||
3/15 p=0.0075 on both. The raw classifier is a clear, decisive learner and a
|
||
clear loser to the Linear base: applying an argmax GF bucket costs ~13 pp early.
|
||
|
||
### Real DrussGT fixtures, bmPoint, seeds=3 (gated)
|
||
|
||
| variant | early | overall |
|
||
|---|---|---|
|
||
| Linear | 7.2% (1480/20498) | 4.7% (11277/241423) |
|
||
| Tsetlin | 7.0% (4403/62617) | 4.8% (34588/724717) |
|
||
| TMPattern | 7.2% (4441/61842) | 4.6% (33341/724556) |
|
||
| TMPatternShuf | 6.6% (4112/61979) | 3.4% (24847/724655) |
|
||
|
||
Online accuracy 50.8%. TMPattern is statistically indistinguishable from Linear
|
||
here (early per-run mean 13.67 vs 13.61; overall 5.70 vs 5.77) and beats its
|
||
control on overall — i.e. on the arrival-time metric the correction is neutral,
|
||
not harmful.
|
||
|
||
### Synthetic fixtures (known rules) — the mechanism works when motion is predictable
|
||
|
||
bmPath, seeds=1, soft readout K=9: Linear early 76.3% / overall 71.4%;
|
||
TMPattern early 76.6% / overall 71.8%; Shuf early 76.6% / overall 67.8%.
|
||
Per-fixture gains vs Linear: wall-bounce 567 vs 537, energy-threshold-turner 332
|
||
vs 319; loss: constant-velocity 417 vs 431.
|
||
|
||
bmPoint, seeds=1, gated hard K=5: Linear 66.4% / 59.6%; TMPattern 66.8% / 60.6%;
|
||
Shuf 55.7% / 50.1%. Energy-threshold-turner 268 vs 212, wall-bounce 585 vs 573.
|
||
|
||
## Verdict
|
||
|
||
* **Learning**: YES, decisively. The discrete-target TM predicts the GF bucket
|
||
far above chance (46% vs 20%) and beats its shuffled control (ungated 18/18,
|
||
p<0.0001). The "regression is a TM mismatch" diagnosis was correct.
|
||
* **Beats Linear**: NO on the real surfers under bmPath (significantly worse,
|
||
p=0.0075). Neutral under bmPoint. Matches/slightly beats Linear only on
|
||
synthetic motion whose future is genuinely predictable.
|
||
* **Best configuration found**: gated hard K=5, `TM_CONF_MARGIN=0.25`,
|
||
`TM_SHRINK=0.5` → 27.9% early / 22.0% overall (bmPath, real), +5.6 pp early /
|
||
+1.7 pp overall vs the default TM gun, but 6.1 pp early / 2.3 pp overall
|
||
behind Linear.
|
||
|
||
## MEASURED vs INFERRED
|
||
|
||
MEASURED: every number in the tables above (pooled hits/shots, per-run
|
||
distributions, paired sign tests, online classification accuracies). The
|
||
position-ring label is our own recorded history at the base arrival tick; the
|
||
shuffled control replaces only the label class with a uniform random draw.
|
||
|
||
INFERRED: that the residual loss on real surfers is because the linear lead is
|
||
already the modal GF (label histogram is centred: real labels
|
||
[3.8,6.6,17.6,6.6,3.5]×10⁵ for 5 classes) and the enemy's per-tick lateral
|
||
reversal sign is not predictable enough from the 40 context bits to make a
|
||
corrective excursion net-positive. Not directly measured.
|
||
|
||
## What to try next (not done, time-boxed out)
|
||
|
||
1. **Radial target instead of angular.** `forecastRadialBlend` work showed the
|
||
dominant surfer error is range-holding (radial), not angle. A TM classifier
|
||
over a RADIAL displacement bucket applied as an aim-distance correction
|
||
targets the error the base actually has room to fix, and should matter most
|
||
under bmPoint.
|
||
2. **Binary reversal with a two-candidate aim** (brief candidate #1, unimplemented):
|
||
predict "will the enemy reverse lateral direction before arrival?" and choose
|
||
between the linear lead and a reversed lead. Same GF family, but a 2-class
|
||
target is far more data-efficient; expected neutral given the GF result.
|
||
3. **Condition a genuinely weaker base.** The measured wall says the deficit is
|
||
the baseline; the Linear base leaves the TM no headroom. Feeding the TM the
|
||
residual of `forecastRadialBlend` (a base that is worse on straight-liners but
|
||
range-correct on surfers) is where a learned correction could plausibly pay.
|
||
4. **Richer context.** 46% accuracy leaves room; the current context lacks the
|
||
enemy's own recent GF history / segmentation that KNN/DecayGF exploit.
|
||
|
||
---
|
||
|
||
# ROUND 2 — fix the base, then try a target Linear cannot predict
|
||
|
||
Date: 2026-09-22. Artifacts: `common_libs/guns/tm_pattern.nim` (extended),
|
||
`common_libs/tests/sweep_tm_pattern.nim` (extended). Raw outputs:
|
||
`/tmp/tm2_real_path_s3.txt`, `/tmp/tm2_real_point_s3.txt`,
|
||
`/tmp/tm_best_path_s3.txt`. All numbers below are MEASURED unless a line says
|
||
INFERRED.
|
||
|
||
Round 1's "best" gun predicted the LATERAL GF bucket. Round 2 adds a RADIAL
|
||
head (aim-distance correction) and a binary REVERSAL head (flip the GF sign),
|
||
both on the same 40-bit context and the same TM core, selected by a runtime
|
||
`targetMode` (`tmGF` | `tmRadial` | `tmReversal`). The shuffled-feedback control
|
||
now randomises only the head the active mode is claiming.
|
||
|
||
## Task 1 — the base is EXACTLY Linear (premise refuted)
|
||
|
||
`tm_pattern`'s base is `forecastLinear`, which already iterates the flight time
|
||
(5-iteration fixed point, same as `LinearGun`). The only deviation from
|
||
`LinearGun` was the wall clamp: the base path clamped to `[BotRadius, W-BotRadius]`
|
||
(17 px inset) instead of `LinearGun`'s `[0, W]`. Added a `forceBase` flag and a
|
||
`TMPatternBase` variant, and made the zero-correction path return `f.x, f.y`
|
||
with the exact `[0, W]` clamp.
|
||
|
||
Real DrussGT fixtures, bmPath, seeds=3, 18 fixture×seed runs, 77 rounds pooled:
|
||
|
||
| variant | early | overall |
|
||
|---|---|---|
|
||
| Linear | 34.0% (6358/18715) | 24.3% (58297/239943) |
|
||
| LinearOldClamp (pre-fix base, BotRadius inset) | 34.2% (6350/18592) | 24.7% (59217/239891) |
|
||
| **TMPatternBase (forceBase, exact Linear clamp)** | **34.0% (6358/18715)** | **24.3% (58297/239943)** |
|
||
|
||
Paired sign test Linear vs TMPatternBase: **18 ties, 0 wins each, p=1.000** on
|
||
both early and overall; every per-run row (hits, shots, per-bin) is byte-for-byte
|
||
identical. Per-run means identical: early 35.52%, overall 26.86%.
|
||
|
||
Verdict: **the base was never behind.** It IS `LinearGun` to the last floating
|
||
point. The earlier "one-shot, non-iterating baseline" finding belonged to the OLD
|
||
`guns/tsetlin.nim`, not to `tm_pattern`. The clamp fix is a wash (the old inset
|
||
was marginally BETTER on overall: 24.7% vs 24.3%), so there is **zero baseline
|
||
headroom** to recover: the entire deficit vs Linear is the TM's corrective
|
||
excursions.
|
||
|
||
## Task 2 — radial target: a structural no-op under bmPath, a real WIN under bmPoint
|
||
|
||
**Structural fact (from `virtual_bullets.nim`, INFERRED then confirmed):** under
|
||
`bmPath` a bullet flies along the aim RAY until it leaves the arena; the aim
|
||
distance only sets `fireDist` (used for the tie-break probe), it does NOT change
|
||
the ray. Moving the aim point radially along the base bearing therefore cannot
|
||
change a `bmPath` hit. Confirmed exactly: on the synthetic set, `TMRadial` vs
|
||
`Linear` scored **8/8 exact ties, p=1.000** under bmPath.
|
||
|
||
Under `bmPoint` the bullet resolves when `travelDist >= fireDist`, so the aim
|
||
distance selects the arrival tick — the radial degree of freedom is live.
|
||
|
||
### bmPath (shipped), real, default config, seeds=3
|
||
|
||
| variant | early | overall |
|
||
|---|---|---|
|
||
| Linear | 34.0% | 24.3% |
|
||
| TMRadial | 33.9% (19038/56203) | 24.1% (173577/719860) |
|
||
| TMRadialShuf | 33.8% (18976/56085) | 24.3% (175154/719781) |
|
||
|
||
Per-run means (n=18): Linear 35.52/26.86; TMRadial 35.44/26.70; Shuf 35.40/26.92.
|
||
Paired sign tests: Linear vs TMRadial early 13/5 p=0.096; overall 15/3 p=0.0075
|
||
(a tiny systematic LOSS, traceable to the `BotRadius` clamp perturbing the ray
|
||
near walls when `radOffset != 0`). TMRadial vs TMRadialShuf early 8/8 p=1.000.
|
||
|
||
**Verdict bmPath: no gain.** Radial is a structural no-op; the shipped metric
|
||
therefore cannot reward Task 2.
|
||
|
||
### bmPoint, real, default config, seeds=3
|
||
|
||
| variant | early | overall |
|
||
|---|---|---|
|
||
| Linear | 7.2% (1480/20498) | 4.7% (11277/241423) |
|
||
| Tsetlin (default gun) | 7.0% (4403/62617) | 4.8% (34588/724717) |
|
||
| **TMRadial** | **9.4% (6013/63785)** | **5.8% (42079/726652)** |
|
||
| TMRadialShuf (control) | 7.0% (4329/61691) | 3.6% (26116/724594) |
|
||
|
||
Per-run means (n=18): Linear 13.61/5.77; Tsetlin 13.13/5.82; TMRadial
|
||
15.32/7.06; Shuf 12.86/4.34. Paired sign tests:
|
||
|
||
* **TMRadial > Linear: early 14/4 p=0.0309; overall 17/1 p=0.0001.**
|
||
* **TMRadial > TMRadialShuf: early 17/1 p=0.0001; overall 18/0 p<0.0001.**
|
||
* TMRadial > Tsetlin: early 17/1 p=0.0001; overall 15/3 p=0.0075.
|
||
* TMRadialShuf vs Linear: early 10/8 p=0.81; overall 12/6 p=0.24 (control sits
|
||
at baseline).
|
||
|
||
Online accuracy of the radial head: 48.8% (511949/1049453) vs **19.9%** shuffled
|
||
chance under bmPoint (46.6% vs 20.0% under bmPath). Radial label histogram
|
||
(raw, seeds=1) = [9058, 5392, 10305, 2236, 1112]: strongly asymmetric — surfers
|
||
are often NEARER than the base constant-velocity prediction at the arrival tick
|
||
(the base overshoots range on range-holders), so class 0 (aim 45–60 px short)
|
||
dominates. That is the mechanism behind the win.
|
||
|
||
Caveat (MEASURED): `labelMiss` is much higher for radial mode (~4.3 M vs ~1.7 M
|
||
for GF) because aiming SHORT resolves the bullet before the base arrival tick,
|
||
so the arrival-tick ring sample is not yet recorded. The radial head is trained
|
||
only on resolvable samples; the win is nonetheless measured on the metric, which
|
||
is label-independent. A deferred-label fix would be the next refinement.
|
||
|
||
## Task 3 — binary reversal: not learnable, and the flip is a no-op
|
||
|
||
Label: net heading turn over the flight opposes the direction the enemy was
|
||
turning at fire time (threshold 10°). Readout: train GF as in round 1, and if
|
||
reversal is predicted, negate the GF correction (`TM_REV_GAIN=1.0`).
|
||
|
||
Label base rate (real, seeds=1, 1 round/fixture): `rev=[24772, 2673]` → the
|
||
positive class is only **9.7%**. The head scores 86.8% (23030/26517) — **below
|
||
the 90.3% majority-class base rate**, i.e. it is not detecting reversals at all,
|
||
only predicting "no reversal". (The shuffled control is 50.2% because its labels
|
||
are balanced.)
|
||
|
||
### bmPath (shipped), real, default config, seeds=3
|
||
|
||
| variant | early | overall |
|
||
|---|---|---|
|
||
| Linear | 34.0% | 24.3% |
|
||
| Tsetlin (round-1 measurement) | 22.3% | 20.3% |
|
||
| TMReversal | 19.5% (11062/56704) | 18.4% (132356/719725) |
|
||
| TMReversalShuf | 19.1% (10925/57063) | 17.8% (128450/720126) |
|
||
|
||
Per-run means: TMReversal 23.45/20.32; Shuf 22.51/20.01. Paired: TMReversal vs
|
||
Shuf early 12/6 p=0.238; overall 8/10 p=0.815 → **no learning effect on hits.**
|
||
|
||
### bmPath, best config (margin=0.25, shrink=0.5), real, seeds=3
|
||
|
||
| variant | early | overall |
|
||
|---|---|---|
|
||
| Linear | 34.0% | 24.3% |
|
||
| TMPattern (gated GF, round-1 best) | 28.3% (15815/55902) | 22.2% (159674/719742) |
|
||
| TMReversal (gated GF + flip) | 28.4% (15944/56067) | 22.3% (160163/719761) |
|
||
| TMReversalShuf | 28.7% (16146/56195) | 21.7% (155973/719841) |
|
||
|
||
Paired: TMPattern vs TMReversal early 11/7 p=0.481, overall 7/11 p=0.481 — the
|
||
flip changes nothing. TMReversal vs Shuf overall 13/5 p=0.096 (not significant).
|
||
|
||
**Verdict: clean negative.** The reversal target as defined is too rare to learn
|
||
(head below the majority baseline), and using it to flip the GF sign is neutral
|
||
to slightly negative on hits. Do not pursue this label; if revisited, balance the
|
||
positive class (per-tick reversal events, or predict the arrival turn direction
|
||
rather than "a reversal happened").
|
||
|
||
## Round-2 overall verdict
|
||
|
||
* On **bmPath (the shipped metric): the TM is NOT competitive with Linear.**
|
||
Base = Linear exactly; radial is a structural no-op; gated GF is significantly
|
||
worse (28.3%/22.2% vs 34.0%/24.3%, p=0.0075); reversal does nothing. The linear
|
||
lead is already the best aim DIRECTION on these surfers and every learned
|
||
angular excursion loses.
|
||
* On **bmPoint: the TM now BEATS Linear and the default Tsetlin gun.**
|
||
`TMRadial` (radial head, `TM_RADIAL_RANGE=60`, `TM_RAD_MARGIN=0.25`, 5 classes,
|
||
gated), 9.4%/5.8% vs Linear 7.2%/4.7% (overall 17/1, p=0.0001) and vs Tsetlin
|
||
7.0%/4.8% (overall 15/3, p=0.0075), with its shuffled control at 7.0%/3.6%.
|
||
This is the first configuration in the whole TM effort that beats both
|
||
baselines with a control-validated margin.
|
||
* **Learning vs controls:** radial head 48.8% vs 19.9% chance (bmPoint); GF head
|
||
reproduces round 1 (48.5% vs ~20%); reversal head does not beat majority.
|
||
|
||
Best configuration if the arrival-time metric is what matters: **TMRadial**. Best
|
||
configuration under the shipped bmPath: **do nothing — keep the Linear base**. The
|
||
evidence says the next step under bmPath is not another bucket target but either a
|
||
richer DIRECTION representation (segmentation / pattern matching, as KNN and
|
||
DecayGF use) or a metric that exposes the radial degree of freedom.
|
||
|
||
Per-enemy specialisation / freshness (MEASURED, unchanged from round 1): each
|
||
offline round is replayed with a FRESH gun instance and the gun calls
|
||
`resetLearning` if the target id changes mid-battle. The offline fixtures are
|
||
single-target, so the mid-battle reset never fires there; its effect is untested
|
||
by these numbers. There is no persistence across battles.
|
||
|
||
## MEASURED vs INFERRED (round 2)
|
||
|
||
* MEASURED: every table, per-run mean, paired sign test, online accuracy, label
|
||
base rate, and the exact `TMPatternBase`/`Linear` byte-for-byte identity.
|
||
* MEASURED: the bmPath radial no-op (synthetic exact ties; real bmPath tiny
|
||
clamp-induced loss).
|
||
* INFERRED: that bmPath ignores radial distance because it flies a ray — read
|
||
from `virtual_bullets.nim`, then confirmed by the synthetic tie.
|
||
* INFERRED: that the radial win comes from surfers being NEARER than the base
|
||
prediction (range-holding), supported by the asymmetric radial label histogram
|
||
but not separately modelled.
|
||
|
||
---
|
||
|
||
# ROUND 3 — ABLATION: does the radial win even need the Tsetlin Machine?
|
||
|
||
Date: 2026-09-22. Artifacts: `common_libs/guns/radial_offset.nim` (new),
|
||
`common_libs/tests/sweep_radial_offset.nim` (new), `common_libs/guns/tm_pattern.nim`
|
||
(label/readout instrumentation only). Raw outputs: `/tmp/ro_point_s3.txt`,
|
||
`/tmp/ro_path_s3.txt` (`--set=real --seeds=3`), `/tmp/ro_point_s1.txt`.
|
||
|
||
## The question
|
||
|
||
Commit 589a230 retracted the radial head's "conditional learning" claim: the
|
||
radial head's online accuracy (57.0%) is AT/BELOW the 58.2% majority baseline, and
|
||
the bmPoint metric win was attributed to a **NET-POSITIVE AVERAGE RADIAL SHIFT**.
|
||
If that is what it is, a FIXED radial shift should reproduce the win with no
|
||
learning, no 0.36 ms/tick cost and no risk.
|
||
|
||
`radial_offset.nim` is exactly that fixed shift and nothing else: the exact
|
||
`forecastLinear` bearing, aim distance `f.dist * scale + offsetPx`, and the SAME
|
||
`[BotRadius, arena-BotRadius]` clamp the TM's corrective path uses. It is
|
||
stateless, so two runs are identical by construction. The sweep runs **Linear**,
|
||
a grid of constants, **TMRadial** and **TMRadialShuf** under BOTH metrics, using
|
||
the identical fixture set / round splitting / per-fixture fresh gun methodology
|
||
as `sweep_tm_pattern.nim`.
|
||
|
||
**Harness sanity check (MEASURED):** `TMRadial` in this report reproduces the
|
||
committed numbers exactly — bmPoint early 9.1% / overall 5.7%, vs Linear early
|
||
16/2 p=0.0013 and overall 18/0 p<0.0001, vs Shuf 18/0 p<0.0001. So the ablation
|
||
runs on the same experiment, not a re-derivation.
|
||
|
||
0 dropped bullets in every run (ring never clobbered); `labelMiss=0` and
|
||
`traceMiss=0` for every TM row (the deferred-label fix holds).
|
||
|
||
## bmPoint (seeds=3)
|
||
|
||
Pooled over 77 rounds for the deterministic arms (one pass; replicated across the
|
||
3 seeds only for the paired test) and 231 rounds for the TM arms (3 seeds).
|
||
|
||
| variant | early | overall |
|
||
|---|---|---|
|
||
| Linear | 7.2% (1480/20498) | 4.7% (11277/241423) |
|
||
| `RO_s1.00` (base + BotRadius clamp only) | 7.2% (1481/20626) | 4.7% (11388/241551) |
|
||
| `RO_s0.98` (scale 0.98) | 8.0% (1669/20781) | 5.8% (14033/241677) |
|
||
| **`RO_s0.95`** | **8.8% (1852/21001)** | **6.3% (15299/241890)** |
|
||
| `RO_o-10` (fixed -10 px) | 8.4% (1756/20790) | 5.9% (14336/241689) |
|
||
| **`RO_o-20`** | 7.9% (1658/20939) | **6.4% (15445/241836)** |
|
||
| `RO_o-30` | 5.4% (1130/21117) | 4.9% (11740/242007) |
|
||
| `RO_o-40` | 4.1% (883/21292) | 3.7% (8989/242166) |
|
||
| `RO_o-60` | 4.7% (1019/21655) | 3.0% (7173/242495) |
|
||
| `RO_o+30` (opposite-direction control) | 0.6% (131/20159) | 0.3% (650/241168) |
|
||
| **TMRadial** | **9.1% (5776/63518)** | 5.7% (41232/726357) |
|
||
| TMRadialShuf | 7.0% (4314/61672) | 3.8% (27317/724473) |
|
||
|
||
Per-run means over the 18 fixture×seed runs:
|
||
Linear 13.61 / 5.77; `RO_s0.95` **14.86 / 7.47**; `RO_o-20` 10.35 / 7.45;
|
||
TMRadial **15.08 / 6.89**; Shuf 12.89 / 4.54.
|
||
|
||
Paired sign tests (exact two-sided binomial, 18 pairs):
|
||
|
||
| A | B | metric | nA>B | nB>A | p |
|
||
|---|---|---|---|---|---|
|
||
| RO_s0.95 | Linear | early | 15 | 3 | **0.0075** |
|
||
| RO_s0.95 | Linear | overall | 15 | 3 | **0.0075** |
|
||
| RO_s0.98 | Linear | overall | 18 | 0 | **<0.0001** |
|
||
| RO_o-20 | Linear | early | 15 | 3 | **0.0075** |
|
||
| RO_o-20 | Linear | overall | 15 | 3 | **0.0075** |
|
||
| RO_o-10 | Linear | overall | 18 | 0 | **<0.0001** |
|
||
| RO_o+30 | Linear | both | 0 | 18 | **<0.0001** (wrong direction) |
|
||
| TMRadial | Linear | early | 16 | 2 | **0.0013** |
|
||
| TMRadial | Linear | overall | 18 | 0 | **<0.0001** |
|
||
| TMRadial | Shuf | both | 18 | 0 | **<0.0001** |
|
||
| **RO_s0.95** | **TMRadial** | **early** | **9** | **9** | **1.0000 (tie)** |
|
||
| **RO_s0.95** | **TMRadial** | **overall** | **15** | **3** | **0.0075 (constant wins)** |
|
||
| RO_o-20 | TMRadial | early | 6 | 12 | 0.2379 |
|
||
| RO_o-20 | TMRadial | overall | 12 | 6 | 0.2379 |
|
||
| RO_s0.98 | TMRadial | overall | 6 | 12 | 0.2379 |
|
||
|
||
**MEASURED conclusion (bmPoint):** a constant radial shift of **scale 0.95**
|
||
(or fixed **−20 px**) *ties the learned radial head on EARLY and beats it on
|
||
OVERALL* (7.47% vs 6.89% per-run mean, 15/3 p=0.0075). The learned head's whole
|
||
nominal advantage is its slightly higher early rate (9.1% vs 8.8% pooled,
|
||
15.08% vs 14.86% per-run), and that is a statistical tie (9/9, p=1.0).
|
||
`RO_o+30` (aiming the opposite way) collapses to 0.3%, confirming the direction of
|
||
the effect is real and not a clamp artefact.
|
||
|
||
## bmPath (the SHIPPED metric) — the radial avenue is a dead end
|
||
|
||
| variant | early | overall |
|
||
|---|---|---|
|
||
| Linear | 34.0% (6358/18715) | 24.3% (58297/239943) |
|
||
| `RO_s1.00` (clamp only) | 34.2% (6350/18592) | **24.7%** (59217/239891) |
|
||
| `RO_s0.98` | 34.2% (6360/18623) | 24.6% (58950/239906) |
|
||
| `RO_s0.95` (best bmPoint constant) | 34.0% (6344/18675) | 24.4% (58530/239937) |
|
||
| `RO_o-20` (best bmPoint constant) | 34.1% (6354/18649) | 24.5% (58699/239921) |
|
||
| `RO_o+30` (best bmPath constant) | 34.4% (6357/18498) | **24.8%** (59366/239852) |
|
||
| `RO_s0.80` | 33.6% (6346/18886) | 23.4% (56151/240035) |
|
||
| TMRadial | 33.8% (19002/56211) | 24.0% (173001/719875) |
|
||
| TMRadialShuf | 33.9% (19026/56076) | 24.3% (175262/719776) |
|
||
|
||
Per-run means (n=18): Linear 35.52 / 26.86; `RO_s1.00` 35.71 / **27.23**;
|
||
`RO_s0.95` 35.59 / 26.97; `RO_o-20` 35.63 / 27.04; **TMRadial 35.37 / 26.63**
|
||
(a loss).
|
||
|
||
Paired sign tests (bmPath):
|
||
|
||
* **TMRadial vs Linear: early 5/13 p=0.0963; overall 2/16 p=0.0013 — a
|
||
systematic LOSS.** (Matches the committed Round-2 finding.)
|
||
* `RO_s1.00` vs Linear: overall 18/0 p<0.0001 (+0.4 pp) — this is the pre-existing
|
||
**BotRadius-clamp** effect, *not* the radial shift.
|
||
* Every constant between 0.98 and 0.95 and every fixed offset −10..−20 is within
|
||
±0.2 pp of Linear; `RO_s0.95` (a real +1.6 pp win on bmPoint overall) is only
|
||
+0.1 pp here and *below* the clamp-only arm. Strong shrink (≤0.90, ≤−40 px) is a
|
||
significant loss (e.g. `RO_s0.80` 3/15 p=0.0075 early, 0/18 overall).
|
||
* `RO_o+30` — the OPPOSITE direction to what bmPoint wants — is the best bmPath
|
||
constant (+0.5 pp, 18/0). So the bmPath response to the radial knob is
|
||
**clamp-mediated and direction-insensitive**, i.e. the radial degree of freedom
|
||
is a structural no-op, exactly as Round 2 argued.
|
||
|
||
**MEASURED conclusion (bmPath):** neither the learned radial head nor any
|
||
constant reproduces a real gain. TMRadial is a systematic *loss*. **The whole
|
||
radial avenue cannot help the shipped configuration.**
|
||
|
||
## The radial-label distribution (MEASURED)
|
||
|
||
Pooled `TMRadial` training labels (3 seeds, n=725997), class centres
|
||
−60/−30/0/+30/+60 px:
|
||
`radLabelHist = [415166, 126461, 120325, 44694, 19351]` → **majority class 57.2%**,
|
||
`meanRadDelta = −82.30 px`, `meanAbsRadDelta = 90.13 px`.
|
||
The head's online accuracy is **56.2%**, i.e. at/below that majority.
|
||
The mean APPLIED shift is **−37.61 px** (`radChosenHist =
|
||
[460960, 35411, 249071, 6832, 2994]`).
|
||
|
||
Metric-free raw base radial error (each fired bullet's enemy radius minus
|
||
`forecastLinear`'s fire distance, at the base arrival tick):
|
||
|
||
| fixture | n | mean px | meanAbs px | frac nearer | frac farther |
|
||
|---|---|---|---|---|---|
|
||
| drussgt_vs_crazy | 35959 | −87.6 | 92.7 | 0.693 | 0.053 |
|
||
| drussgt_vs_spinbot | 19830 | −100.4 | 107.7 | 0.769 | 0.062 |
|
||
| drussgt_vs_drussgt | 26060 | −70.8 | 85.8 | 0.627 | 0.142 |
|
||
| tr_drussgt_vs_crazy | 45905 | −99.0 | 104.4 | 0.805 | 0.053 |
|
||
| tr_drussgt_vs_spinbot | 43150 | −92.4 | 96.7 | 0.793 | 0.044 |
|
||
| tr_drussgt_vs_modularbot | 79970 | −79.9 | 90.7 | 0.705 | 0.108 |
|
||
|
||
The enemy is NEARER than the constant-velocity prediction in **63–81%** of fired
|
||
bullets and FARTHER in only **4–14%**, consistently across all six captures and
|
||
both metrics. So the net-short bias is a GENUINE property of these range-holding
|
||
surfers against the constant-velocity base (the base lets range grow
|
||
geometrically; they hold it) — not a one-fixture artefact.
|
||
|
||
**INFERRED:** the mean label (−82 px) is much larger than the OPTIMAL constant
|
||
shift (−20 px). The mean is dominated by large radial errors that miss regardless
|
||
of the shift; the near-miss window is served better by a small shift, and a
|
||
larger shift also resolves the bullet at an earlier tick. The exact reason a
|
||
−20 px shift beats −60 px is not separately modelled.
|
||
|
||
## Does the optimal constant vary by adversary? (MEASURED)
|
||
|
||
bmPoint, best constant PER FIXTURE (in-sample upper bound), with Linear and
|
||
TMRadial for reference:
|
||
|
||
| fixture | best early | best overall | Linear overall | TMRadial overall |
|
||
|---|---|---|---|---|
|
||
| drussgt_vs_crazy | RO_s0.80 (10.1%) | RO_o-10 (16.9%) | 15.8% | 17.0% |
|
||
| drussgt_vs_spinbot | RO_s0.95 (9.9%) | RO_s0.95 (11.5%) | 7.5% | 9.9% |
|
||
| drussgt_vs_drussgt | RO_s1.00 (55.3%) | RO_s0.98 (4.3%) | 3.9% | 4.1% |
|
||
| tr_drussgt_vs_crazy | RO_s0.85 (9.5%) | RO_o-30 (5.8%) | 2.9% | 4.4% |
|
||
| tr_drussgt_vs_modularbot | RO_s0.95 (6.8%) | RO_s0.95 (2.7%) | 1.3% | 2.1% |
|
||
| tr_drussgt_vs_spinbot | RO_s0.80 (6.8%) | RO_s0.95 (7.4%) | 3.3% | 3.7% |
|
||
|
||
**MEASURED:** the per-fixture optimum DOES vary (fixed −10 for `crazy`, −30 for
|
||
`tr_crazy`, scale 0.95 for three others). **But** one GLOBAL constant (0.95)
|
||
still beats TMRadial on pooled and per-run overall bmPoint. So the per-adversary
|
||
variation is not enough to justify the learned head: a fixed 0.95 is already the
|
||
best pooled point-metric arm measured here.
|
||
|
||
## DIRECT VERDICT
|
||
|
||
1. **bmPoint: REPLACE the radial TM with a constant.** The best constant
|
||
(scale 0.95, equivalently fixed −20 px) statistically TIES the TM on early
|
||
(9/9 p=1.0) and BEATS it on overall (15/3 p=0.0075; 7.47% vs 6.89% per-run
|
||
mean). The head never out-classifies its majority baseline (56.2% vs 57.2%)
|
||
and its mean applied shift (−37.6 px) is roughly twice the optimal constant.
|
||
The TM buys nothing a constant does not, and costs 0.36 ms/tick + complexity.
|
||
2. **bmPath (shipped): DEAD END.** No constant and no TM improves it; TMRadial
|
||
is a systematic loss (2/16 p=0.0013), and the only real bmPath effect in the
|
||
table is the BotRadius clamp, which is direction-insensitive. Drop the radial
|
||
mode from any shipped configuration.
|
||
3. **Fragility argument fails.** The optimum does vary per adversary, but a
|
||
single global constant already matches/beats the adaptively-trained head — so
|
||
the TM is not earning its cost even by the "per-adversary adaptation"
|
||
argument (it is cold-every-battle and trains online within the battle, yet
|
||
still loses to the global 0.95).
|
||
|
||
Net: **do not keep the radial TM.** If the arrival metric ever matters, ship the
|
||
stateless constant; for the current shipped metric, the radial mode (and its
|
||
registered gun id 14) is not justified.
|
||
|
||
## MEASURED vs INFERRED (round 3)
|
||
|
||
* MEASURED: every table, pooled rate, per-run mean, paired sign test, label
|
||
histogram, mean/abs radial delta, applied-shift mean, online accuracy, and the
|
||
exact reproduction of the committed TMRadial numbers.
|
||
* MEASURED: the constant-offset arm is stateless (verbatim `forecastLinear`
|
||
bearing, `f.dist*scale+offsetPx`, TM corrective clamp), so its rollout is
|
||
deterministic and its replicated per-seed values are legitimate.
|
||
* MEASURED: `RO_s1.00` isolates the BotRadius clamp — on bmPoint it is identical
|
||
to Linear (7.2/4.7%), on bmPath it is the +0.4 pp arm; the radial shift itself
|
||
adds nothing on bmPath.
|
||
* INFERRED: the explanation of the label-mean (−82 px) vs optimal shift (−20 px)
|
||
gap (large-error tail + earlier resolution tick); the direction claim itself is
|
||
measured (the +30 control collapses).
|
||
* INFERRED (not measured): whether a per-adversary constant would beat a global
|
||
one out-of-sample — the per-fixture optima above are in-sample.
|
||
|
||
## How to reproduce (round 3)
|
||
|
||
```
|
||
nim c --path:common_libs -d:release -o:/tmp/sweep_radial_offset \
|
||
common_libs/tests/sweep_radial_offset.nim
|
||
/tmp/sweep_radial_offset --set=real --metric=point --seeds=3
|
||
/tmp/sweep_radial_offset --set=real --metric=path --seeds=3
|
||
# smoke: --seeds=1
|
||
```
|
||
|