SirStone/SirRoboGarage
1
0
Fork 0
Code Issues 28 Pull Requests Actions Projects Releases Wiki Activity

TM gun round 2: base was never behind; RADIAL target beats Linear on bmPoint

Browse Source 
=== TASK 1: MY PREMISE WAS REFUTED ===
I instructed the job to "fix the baseline" because an earlier measurement said the
TM gun's base did not iterate flight time like `LinearGun`. MEASURED: the new gun's
base is BYTE-FOR-BYTE `LinearGun` - 18/18 runs tie exactly, p=1.000, every per-run
row byte-identical. The "non-iterating baseline" belonged to the OLD `tsetlin.nim`,
not this gun. So no fix was needed, and the earlier inference should not have been
generalised to the new gun. (It did still align the zero-correction clamp to
LinearGun's exact [0, arena] range, and reports the old BotRadius-inset base was a
wash/marginally better at 34.2%/24.7%.)

=== TASK 2: THE RADIAL TARGET - A CONTROL-VALIDATED WIN, BUT ONLY ON bmPoint ===
Instead of the lateral (GF-bucket) component - which the linear lead already
captures - the TM now predicts the RADIAL component: will the enemy be nearer or
farther than the base prediction when our bullet arrives? A 5-class radial head
sharing the same 40-bit context and TM core; the readout advances/retards the aim
distance along the base bearing.

  under bmPath (the SHIPPED metric): STRUCTURAL NO-OP
    synthetic 8/8 exact ties, p=1.0; real 33.9%/24.1% vs Linear 34.0%/24.3%
  under bmPoint: A WIN, control-validated
    TMRadial 9.4% (6013/63785) / 5.8% (42079/726652)
    Linear   7.2% / 4.7%          overall 17/1, p=0.0001
    Tsetlin  7.0% / 4.8%          overall 15/3, p=0.0075
    shuffled 7.0% / 3.6%          early 17/1 p=0.0001; overall 18/0, p<0.0001
  radial head online accuracy 48.8% vs 19.9% shuffled chance and 36.7% majority
  -> it is CONDITIONAL learning, not a constant short-range bias.
Best config: TM_RADIAL_RANGE=60, TM_RAD_MARGIN=0.25, 5 classes.

CAVEAT THAT MATTERS: a win on `bmPoint` is NOT yet evidence of a real win. `bmPath`
is the shipped SELECTION metric precisely because it beat `bmPoint` on real hit
rate (7.43% vs 4.70%). But that A/B was about which gun to PICK, not about gun
QUALITY - a gun can be better in reality while scoring worse on the selection
metric. So this needs a LIVE test, and it is the decisive one.

=== TASK 3: REVERSAL TARGET - CLEAN NEGATIVE ===
The label positive rate is only 9.7% (rev=[24772,2673]) and the head's 86.8%
accuracy is BELOW the 90.3% majority baseline: it does not learn the positive
class at all. Hit-rate effect neutral (bmPath 19.5%/18.4% vs shuffled 19.1%/17.8%,
p=0.24/0.82). Dropped.

=== OVERALL ===
Not competitive on the shipped bmPath metric (gated GF 28.3%/22.2% vs Linear
34.0%/24.3%, p=0.0075). Better than Linear on bmPoint via TMRadial (+2.2pp early,
+1.1pp overall). Per-enemy reset exists; a fresh gun per round; NO cross-battle
persistence (the user's non-negotiable).

MEASURED LIMITATION: radial mode has a high labelMiss because aiming short
resolves BEFORE the base arrival tick, biasing training toward resolvable samples.
The metric win is label-independent. A deferred-label fix is the next refinement.
INFERRED: the mechanism is surfers being NEARER than the base prediction
(range-holding); a constant-short-offset ablation would separate a learned
short-range bias from genuine per-tick conditional prediction.
This commit is contained in:
Davide Cappellini
2026-09-22 01:27:59 +02:00
parent 78975a35c4
commit 1ea72c7f14
3 changed files with 435 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files
common_libs/tests/tm_pattern_sweep_results.md
+185
View File
@@ -179,3 +179,188 @@ corrective excursion net-positive. Not directly measured.
range-correct on surfers) is where a learned correction could plausibly pay.
4. **Richer context.** 46% accuracy leaves room; the current context lacks the
enemy's own recent GF history / segmentation that KNN/DecayGF exploit.
---
# ROUND 2 — fix the base, then try a target Linear cannot predict
Date: 2026-09-22. Artifacts: `common_libs/guns/tm_pattern.nim` (extended),
`common_libs/tests/sweep_tm_pattern.nim` (extended). Raw outputs:
`/tmp/tm2_real_path_s3.txt`, `/tmp/tm2_real_point_s3.txt`,
`/tmp/tm_best_path_s3.txt`. All numbers below are MEASURED unless a line says
INFERRED.
Round 1's "best" gun predicted the LATERAL GF bucket. Round 2 adds a RADIAL
head (aim-distance correction) and a binary REVERSAL head (flip the GF sign),
both on the same 40-bit context and the same TM core, selected by a runtime
`targetMode` (`tmGF` | `tmRadial` | `tmReversal`). The shuffled-feedback control
now randomises only the head the active mode is claiming.
## Task 1 — the base is EXACTLY Linear (premise refuted)
`tm_pattern`'s base is `forecastLinear`, which already iterates the flight time
(5-iteration fixed point, same as `LinearGun`). The only deviation from
`LinearGun` was the wall clamp: the base path clamped to `[BotRadius, W-BotRadius]`
(17 px inset) instead of `LinearGun`'s `[0, W]`. Added a `forceBase` flag and a
`TMPatternBase` variant, and made the zero-correction path return `f.x, f.y`
with the exact `[0, W]` clamp.
Real DrussGT fixtures, bmPath, seeds=3, 18 fixture×seed runs, 77 rounds pooled:
| variant | early | overall |
|---|---|---|
| Linear | 34.0% (6358/18715) | 24.3% (58297/239943) |
| LinearOldClamp (pre-fix base, BotRadius inset) | 34.2% (6350/18592) | 24.7% (59217/239891) |
| **TMPatternBase (forceBase, exact Linear clamp)** | **34.0% (6358/18715)** | **24.3% (58297/239943)** |
Paired sign test Linear vs TMPatternBase: **18 ties, 0 wins each, p=1.000** on
both early and overall; every per-run row (hits, shots, per-bin) is byte-for-byte
identical. Per-run means identical: early 35.52%, overall 26.86%.
Verdict: **the base was never behind.** It IS `LinearGun` to the last floating
point. The earlier "one-shot, non-iterating baseline" finding belonged to the OLD
`guns/tsetlin.nim`, not to `tm_pattern`. The clamp fix is a wash (the old inset
was marginally BETTER on overall: 24.7% vs 24.3%), so there is **zero baseline
headroom** to recover: the entire deficit vs Linear is the TM's corrective
excursions.
## Task 2 — radial target: a structural no-op under bmPath, a real WIN under bmPoint
**Structural fact (from `virtual_bullets.nim`, INFERRED then confirmed):** under
`bmPath` a bullet flies along the aim RAY until it leaves the arena; the aim
distance only sets `fireDist` (used for the tie-break probe), it does NOT change
the ray. Moving the aim point radially along the base bearing therefore cannot
change a `bmPath` hit. Confirmed exactly: on the synthetic set, `TMRadial` vs
`Linear` scored **8/8 exact ties, p=1.000** under bmPath.
Under `bmPoint` the bullet resolves when `travelDist >= fireDist`, so the aim
distance selects the arrival tick — the radial degree of freedom is live.
### bmPath (shipped), real, default config, seeds=3
| variant | early | overall |
|---|---|---|
| Linear | 34.0% | 24.3% |
| TMRadial | 33.9% (19038/56203) | 24.1% (173577/719860) |
| TMRadialShuf | 33.8% (18976/56085) | 24.3% (175154/719781) |
Per-run means (n=18): Linear 35.52/26.86; TMRadial 35.44/26.70; Shuf 35.40/26.92.
Paired sign tests: Linear vs TMRadial early 13/5 p=0.096; overall 15/3 p=0.0075
(a tiny systematic LOSS, traceable to the `BotRadius` clamp perturbing the ray
near walls when `radOffset != 0`). TMRadial vs TMRadialShuf early 8/8 p=1.000.
**Verdict bmPath: no gain.** Radial is a structural no-op; the shipped metric
therefore cannot reward Task 2.
### bmPoint, real, default config, seeds=3
| variant | early | overall |
|---|---|---|
| Linear | 7.2% (1480/20498) | 4.7% (11277/241423) |
| Tsetlin (default gun) | 7.0% (4403/62617) | 4.8% (34588/724717) |
| **TMRadial** | **9.4% (6013/63785)** | **5.8% (42079/726652)** |
| TMRadialShuf (control) | 7.0% (4329/61691) | 3.6% (26116/724594) |
Per-run means (n=18): Linear 13.61/5.77; Tsetlin 13.13/5.82; TMRadial
15.32/7.06; Shuf 12.86/4.34. Paired sign tests:
* **TMRadial > Linear: early 14/4 p=0.0309; overall 17/1 p=0.0001.**
* **TMRadial > TMRadialShuf: early 17/1 p=0.0001; overall 18/0 p<0.0001.**
* TMRadial > Tsetlin: early 17/1 p=0.0001; overall 15/3 p=0.0075.
* TMRadialShuf vs Linear: early 10/8 p=0.81; overall 12/6 p=0.24 (control sits
at baseline).
Online accuracy of the radial head: 48.8% (511949/1049453) vs **19.9%** shuffled
chance under bmPoint (46.6% vs 20.0% under bmPath). Radial label histogram
(raw, seeds=1) = [9058, 5392, 10305, 2236, 1112]: strongly asymmetric — surfers
are often NEARER than the base constant-velocity prediction at the arrival tick
(the base overshoots range on range-holders), so class 0 (aim 45–60 px short)
dominates. That is the mechanism behind the win.
Caveat (MEASURED): `labelMiss` is much higher for radial mode (~4.3 M vs ~1.7 M
for GF) because aiming SHORT resolves the bullet before the base arrival tick,
so the arrival-tick ring sample is not yet recorded. The radial head is trained
only on resolvable samples; the win is nonetheless measured on the metric, which
is label-independent. A deferred-label fix would be the next refinement.
## Task 3 — binary reversal: not learnable, and the flip is a no-op
Label: net heading turn over the flight opposes the direction the enemy was
turning at fire time (threshold 10°). Readout: train GF as in round 1, and if
reversal is predicted, negate the GF correction (`TM_REV_GAIN=1.0`).
Label base rate (real, seeds=1, 1 round/fixture): `rev=[24772, 2673]` → the
positive class is only **9.7%**. The head scores 86.8% (23030/26517) — **below
the 90.3% majority-class base rate**, i.e. it is not detecting reversals at all,
only predicting "no reversal". (The shuffled control is 50.2% because its labels
are balanced.)
### bmPath (shipped), real, default config, seeds=3
| variant | early | overall |
|---|---|---|
| Linear | 34.0% | 24.3% |
| Tsetlin (round-1 measurement) | 22.3% | 20.3% |
| TMReversal | 19.5% (11062/56704) | 18.4% (132356/719725) |
| TMReversalShuf | 19.1% (10925/57063) | 17.8% (128450/720126) |
Per-run means: TMReversal 23.45/20.32; Shuf 22.51/20.01. Paired: TMReversal vs
Shuf early 12/6 p=0.238; overall 8/10 p=0.815 → **no learning effect on hits.**
### bmPath, best config (margin=0.25, shrink=0.5), real, seeds=3
| variant | early | overall |
|---|---|---|
| Linear | 34.0% | 24.3% |
| TMPattern (gated GF, round-1 best) | 28.3% (15815/55902) | 22.2% (159674/719742) |
| TMReversal (gated GF + flip) | 28.4% (15944/56067) | 22.3% (160163/719761) |
| TMReversalShuf | 28.7% (16146/56195) | 21.7% (155973/719841) |
Paired: TMPattern vs TMReversal early 11/7 p=0.481, overall 7/11 p=0.481 — the
flip changes nothing. TMReversal vs Shuf overall 13/5 p=0.096 (not significant).
**Verdict: clean negative.** The reversal target as defined is too rare to learn
(head below the majority baseline), and using it to flip the GF sign is neutral
to slightly negative on hits. Do not pursue this label; if revisited, balance the
positive class (per-tick reversal events, or predict the arrival turn direction
rather than "a reversal happened").
## Round-2 overall verdict
* On **bmPath (the shipped metric): the TM is NOT competitive with Linear.**
Base = Linear exactly; radial is a structural no-op; gated GF is significantly
worse (28.3%/22.2% vs 34.0%/24.3%, p=0.0075); reversal does nothing. The linear
lead is already the best aim DIRECTION on these surfers and every learned
angular excursion loses.
* On **bmPoint: the TM now BEATS Linear and the default Tsetlin gun.**
`TMRadial` (radial head, `TM_RADIAL_RANGE=60`, `TM_RAD_MARGIN=0.25`, 5 classes,
gated), 9.4%/5.8% vs Linear 7.2%/4.7% (overall 17/1, p=0.0001) and vs Tsetlin
7.0%/4.8% (overall 15/3, p=0.0075), with its shuffled control at 7.0%/3.6%.
This is the first configuration in the whole TM effort that beats both
baselines with a control-validated margin.
* **Learning vs controls:** radial head 48.8% vs 19.9% chance (bmPoint); GF head
reproduces round 1 (48.5% vs ~20%); reversal head does not beat majority.
Best configuration if the arrival-time metric is what matters: **TMRadial**. Best
configuration under the shipped bmPath: **do nothing — keep the Linear base**. The
evidence says the next step under bmPath is not another bucket target but either a
richer DIRECTION representation (segmentation / pattern matching, as KNN and
DecayGF use) or a metric that exposes the radial degree of freedom.
Per-enemy specialisation / freshness (MEASURED, unchanged from round 1): each
offline round is replayed with a FRESH gun instance and the gun calls
`resetLearning` if the target id changes mid-battle. The offline fixtures are
single-target, so the mid-battle reset never fires there; its effect is untested
by these numbers. There is no persistence across battles.
## MEASURED vs INFERRED (round 2)
* MEASURED: every table, per-run mean, paired sign test, online accuracy, label
base rate, and the exact `TMPatternBase`/`Linear` byte-for-byte identity.
* MEASURED: the bmPath radial no-op (synthetic exact ties; real bmPath tiny
clamp-induced loss).
* INFERRED: that bmPath ignores radial distance because it flies a ray — read
from `virtual_bullets.nim`, then confirmed by the synthetic tie.
* INFERRED: that the radial win comes from surfers being NEARER than the base
prediction (range-holding), supported by the asymmetric radial label histogram
but not separately modelled.