TM gun round 2: base was never behind; RADIAL target beats Linear on bmPoint
=== TASK 1: MY PREMISE WAS REFUTED ===
I instructed the job to "fix the baseline" because an earlier measurement said the
TM gun's base did not iterate flight time like `LinearGun`. MEASURED: the new gun's
base is BYTE-FOR-BYTE `LinearGun` - 18/18 runs tie exactly, p=1.000, every per-run
row byte-identical. The "non-iterating baseline" belonged to the OLD `tsetlin.nim`,
not this gun. So no fix was needed, and the earlier inference should not have been
generalised to the new gun. (It did still align the zero-correction clamp to
LinearGun's exact [0, arena] range, and reports the old BotRadius-inset base was a
wash/marginally better at 34.2%/24.7%.)
=== TASK 2: THE RADIAL TARGET - A CONTROL-VALIDATED WIN, BUT ONLY ON bmPoint ===
Instead of the lateral (GF-bucket) component - which the linear lead already
captures - the TM now predicts the RADIAL component: will the enemy be nearer or
farther than the base prediction when our bullet arrives? A 5-class radial head
sharing the same 40-bit context and TM core; the readout advances/retards the aim
distance along the base bearing.
under bmPath (the SHIPPED metric): STRUCTURAL NO-OP
synthetic 8/8 exact ties, p=1.0; real 33.9%/24.1% vs Linear 34.0%/24.3%
under bmPoint: A WIN, control-validated
TMRadial 9.4% (6013/63785) / 5.8% (42079/726652)
Linear 7.2% / 4.7% overall 17/1, p=0.0001
Tsetlin 7.0% / 4.8% overall 15/3, p=0.0075
shuffled 7.0% / 3.6% early 17/1 p=0.0001; overall 18/0, p<0.0001
radial head online accuracy 48.8% vs 19.9% shuffled chance and 36.7% majority
-> it is CONDITIONAL learning, not a constant short-range bias.
Best config: TM_RADIAL_RANGE=60, TM_RAD_MARGIN=0.25, 5 classes.
CAVEAT THAT MATTERS: a win on `bmPoint` is NOT yet evidence of a real win. `bmPath`
is the shipped SELECTION metric precisely because it beat `bmPoint` on real hit
rate (7.43% vs 4.70%). But that A/B was about which gun to PICK, not about gun
QUALITY - a gun can be better in reality while scoring worse on the selection
metric. So this needs a LIVE test, and it is the decisive one.
=== TASK 3: REVERSAL TARGET - CLEAN NEGATIVE ===
The label positive rate is only 9.7% (rev=[24772,2673]) and the head's 86.8%
accuracy is BELOW the 90.3% majority baseline: it does not learn the positive
class at all. Hit-rate effect neutral (bmPath 19.5%/18.4% vs shuffled 19.1%/17.8%,
p=0.24/0.82). Dropped.
=== OVERALL ===
Not competitive on the shipped bmPath metric (gated GF 28.3%/22.2% vs Linear
34.0%/24.3%, p=0.0075). Better than Linear on bmPoint via TMRadial (+2.2pp early,
+1.1pp overall). Per-enemy reset exists; a fresh gun per round; NO cross-battle
persistence (the user's non-negotiable).
MEASURED LIMITATION: radial mode has a high labelMiss because aiming short
resolves BEFORE the base arrival tick, biasing training toward resolvable samples.
The metric win is label-independent. A deferred-label fix is the next refinement.
INFERRED: the mechanism is surfers being NEARER than the base prediction
(range-holding); a constant-short-offset ablation would separate a learned
short-range bias from genuine per-tick conditional prediction.
This commit is contained in:
@@ -24,6 +24,17 @@ import gun_harness/offline_range
|
||||
import range_guns
|
||||
import guns/tm_pattern
|
||||
import guns/linear
|
||||
import guns/lead_forecast
|
||||
|
||||
type
|
||||
LinearInsetGun = object ## pre-fix TMPattern base: same forecast, BotRadius clamp
|
||||
|
||||
proc predict*(g: var LinearInsetGun, state: WorldState, bulletSpeed: float): GunPrediction =
|
||||
let f = forecastLinear(state, bulletSpeed)
|
||||
GunPrediction(x: clamp(f.x, BotRadius, state.arenaWidth - BotRadius),
|
||||
y: clamp(f.y, BotRadius, state.arenaHeight - BotRadius))
|
||||
|
||||
proc onResult*(g: var LinearInsetGun, e: FeedbackEvent) = discard
|
||||
|
||||
const repoRoot = currentSourcePath().parentDir.parentDir.parentDir
|
||||
const fixturesDir = repoRoot / "tools" / "fixtures"
|
||||
@@ -33,6 +44,7 @@ type
|
||||
obs, labelMiss, traceMiss: int
|
||||
labHist, choHist: array[TM_CLASSES, int]
|
||||
classCorrect, classTotal: int
|
||||
radCorrect, radTotal, revCorrect, revTotal: int
|
||||
|
||||
Adapt = object
|
||||
h100, n100, h300, n300, hall, nall, f100, m100: int
|
||||
@@ -47,14 +59,21 @@ type
|
||||
r: Adapt
|
||||
|
||||
VariantKind = enum
|
||||
vLinear, vTsetlin, vTmpat, vTmpatShuf
|
||||
vLinear, vLinearOld, vTsetlin, vTmpat, vTmpatShuf, vTmpatBase,
|
||||
vTmpatRad, vTmpatRadShuf, vTmpatRev, vTmpatRevShuf
|
||||
|
||||
proc variantName(v: VariantKind): string =
|
||||
case v
|
||||
of vLinear: "Linear"
|
||||
of vLinearOld: "LinearOldClamp"
|
||||
of vTsetlin: "Tsetlin"
|
||||
of vTmpat: "TMPattern"
|
||||
of vTmpatShuf: "TMPatternShuf"
|
||||
of vTmpatBase: "TMPatternBase"
|
||||
of vTmpatRad: "TMRadial"
|
||||
of vTmpatRadShuf: "TMRadialShuf"
|
||||
of vTmpatRev: "TMReversal"
|
||||
of vTmpatRevShuf: "TMReversalShuf"
|
||||
|
||||
proc loadRounds(path: string): seq[RoundSpan] =
|
||||
let dir = path.parentDir
|
||||
@@ -77,6 +96,10 @@ proc addAdapt(dst: var Adapt, src: Adapt) =
|
||||
dst.st.traceMiss += src.st.traceMiss
|
||||
dst.st.classCorrect += src.st.classCorrect
|
||||
dst.st.classTotal += src.st.classTotal
|
||||
dst.st.radCorrect += src.st.radCorrect
|
||||
dst.st.radTotal += src.st.radTotal
|
||||
dst.st.revCorrect += src.st.revCorrect
|
||||
dst.st.revTotal += src.st.revTotal
|
||||
for c in 0..<TM_CLASSES:
|
||||
dst.st.labHist[c] += src.st.labHist[c]
|
||||
dst.st.choHist[c] += src.st.choHist[c]
|
||||
@@ -128,6 +151,10 @@ proc replayRound(states: seq[WorldState], lastSeen: seq[int], enemyId, baseTick:
|
||||
res.st.traceMiss = after.traceMiss - obsBefore.traceMiss
|
||||
res.st.classCorrect = after.classCorrect - obsBefore.classCorrect
|
||||
res.st.classTotal = after.classTotal - obsBefore.classTotal
|
||||
res.st.radCorrect = after.radCorrect - obsBefore.radCorrect
|
||||
res.st.radTotal = after.radTotal - obsBefore.radTotal
|
||||
res.st.revCorrect = after.revCorrect - obsBefore.revCorrect
|
||||
res.st.revTotal = after.revTotal - obsBefore.revTotal
|
||||
for c in 0..<TM_CLASSES:
|
||||
res.st.labHist[c] = after.labHist[c] - obsBefore.labHist[c]
|
||||
res.st.choHist[c] = after.choHist[c] - obsBefore.choHist[c]
|
||||
@@ -157,15 +184,18 @@ proc replayFixture(fx: Fixture, path: string, driver: GunDriver, metric: BulletM
|
||||
|
||||
proc emptyStats(): GunStats = GunStats()
|
||||
|
||||
proc makeTmpatDriver(seed: int, shuffle: bool):
|
||||
proc makeTmpatDriver(seed: int, shuffle: bool, forceBase = false,
|
||||
mode = tmGF):
|
||||
tuple[driver: GunDriver, gun: ref TmPatternGun] =
|
||||
let g = new(TmPatternGun)
|
||||
g[] = initTmPatternGun()
|
||||
g[].shuffleLabels = shuffle
|
||||
g[].forceBase = forceBase
|
||||
g[].targetMode = mode
|
||||
if seed >= 0: randomize(seed)
|
||||
result.gun = g
|
||||
result.driver = GunDriver(
|
||||
name: (if shuffle: "TMPatternShuf" else: "TMPattern"),
|
||||
name: (if forceBase: "TMPatternBase" elif shuffle: "TMPatternShuf" else: "TMPattern"),
|
||||
predictCb: proc(state: WorldState, bulletSpeed: float): GunPrediction =
|
||||
g[].predict(state, bulletSpeed),
|
||||
resultCb: proc(e: FeedbackEvent) = g[].onResult(e),
|
||||
@@ -179,6 +209,10 @@ proc tmpatStats(g: ref TmPatternGun): GunStats =
|
||||
result.choHist = g[].chosenHist
|
||||
result.classCorrect = g[].classCorrect
|
||||
result.classTotal = g[].classTotal
|
||||
result.radCorrect = g[].radCorrect
|
||||
result.radTotal = g[].radTotal
|
||||
result.revCorrect = g[].revCorrect
|
||||
result.revTotal = g[].revTotal
|
||||
|
||||
proc fixtureSet(name: string): seq[string] =
|
||||
case name
|
||||
@@ -235,9 +269,15 @@ proc main() =
|
||||
for tok in a[11..^1].split(','):
|
||||
case tok.strip()
|
||||
of "linear": variants.add vLinear
|
||||
of "linear_old": variants.add vLinearOld
|
||||
of "tsetlin": variants.add vTsetlin
|
||||
of "tmpat": variants.add vTmpat
|
||||
of "tmpat_shuf": variants.add vTmpatShuf
|
||||
of "tmbase": variants.add vTmpatBase
|
||||
of "tmrad": variants.add vTmpatRad
|
||||
of "tmrad_shuf": variants.add vTmpatRadShuf
|
||||
of "tmrev": variants.add vTmpatRev
|
||||
of "tmrev_shuf": variants.add vTmpatRevShuf
|
||||
else: discard
|
||||
let metric = if metricName == "point": bmPoint else: bmPath
|
||||
let names = fixtureSet(set)
|
||||
@@ -245,11 +285,13 @@ proc main() =
|
||||
echo "variant,fixture,seed,h100,n100,h300,n300,hall,nall,f100,m100,rounds,obs,labelMiss,traceMiss"
|
||||
|
||||
var rows: seq[Row]
|
||||
var revLabTab = initTable[string, array[2, int]]()
|
||||
var radLabTab = initTable[string, array[TM_CLASSES, int]]()
|
||||
for name in names:
|
||||
let (fx, path) = resolve(name)
|
||||
let fxName = path.extractFilename.replace(".jsonl", "")
|
||||
for v in variants:
|
||||
let nIter = if v == vLinear: 1 else: nSeeds
|
||||
let nIter = if v in [vLinear, vLinearOld, vTmpatBase]: 1 else: nSeeds
|
||||
for seed in 1..nIter:
|
||||
var drv: GunDriver
|
||||
var gun: ref TmPatternGun
|
||||
@@ -257,16 +299,32 @@ proc main() =
|
||||
case v
|
||||
of vLinear:
|
||||
drv = makeDriver("Linear", LinearGun())
|
||||
of vLinearOld:
|
||||
drv = makeDriver("LinearOldClamp", LinearInsetGun())
|
||||
of vTsetlin:
|
||||
let pair = makeTsetlinDriver(seed = seed)
|
||||
drv = pair.driver
|
||||
of vTmpat, vTmpatShuf:
|
||||
let pair = makeTmpatDriver(seed = seed, shuffle = (v == vTmpatShuf))
|
||||
of vTmpat, vTmpatShuf, vTmpatBase, vTmpatRad, vTmpatRadShuf,
|
||||
vTmpatRev, vTmpatRevShuf:
|
||||
let mode =
|
||||
case v
|
||||
of vTmpatRad, vTmpatRadShuf: tmRadial
|
||||
of vTmpatRev, vTmpatRevShuf: tmReversal
|
||||
else: tmGF
|
||||
let shuf = v in [vTmpatShuf, vTmpatRadShuf, vTmpatRevShuf]
|
||||
let pair = makeTmpatDriver(seed = seed, shuffle = shuf,
|
||||
forceBase = (v == vTmpatBase), mode = mode)
|
||||
drv = pair.driver
|
||||
gun = pair.gun
|
||||
obsCount = proc(): GunStats = tmpatStats(gun)
|
||||
let r = replayFixture(fx, path, drv, metric, emptyStats(), obsCount, maxRounds)
|
||||
rows.add Row(variant: variantName(v), fixture: fxName, seed: seed, r: r)
|
||||
if gun != nil:
|
||||
if variantName(v) notin revLabTab: revLabTab[variantName(v)] = [0, 0]
|
||||
if variantName(v) notin radLabTab:
|
||||
radLabTab[variantName(v)] = default(array[TM_CLASSES, int])
|
||||
for c in 0..<2: revLabTab[variantName(v)][c] += gun[].revLabelHist[c]
|
||||
for c in 0..<TM_CLASSES: radLabTab[variantName(v)][c] += gun[].radLabelHist[c]
|
||||
echo &"{variantName(v)},{fxName},{seed},{r.h100},{r.n100},{r.h300},{r.n300}," &
|
||||
&"{r.hall},{r.nall},{r.f100},{r.m100},{r.rounds},{r.st.obs},{r.st.labelMiss},{r.st.traceMiss}"
|
||||
|
||||
@@ -282,8 +340,8 @@ proc main() =
|
||||
&"{a.h300},{a.n300},{rateStr(a.h300, a.n300)},{a.hall},{a.nall}," &
|
||||
&"{rateStr(a.hall, a.nall)},{a.st.obs},{a.st.labelMiss},{a.st.traceMiss}"
|
||||
|
||||
# ── label vs chosen class histogram (TMPattern only) ──
|
||||
for v in [vTmpat, vTmpatShuf]:
|
||||
# ── label vs chosen class histogram (all TM variants) ──
|
||||
for v in [vTmpat, vTmpatShuf, vTmpatRad, vTmpatRadShuf, vTmpatRev, vTmpatRevShuf]:
|
||||
if v in variants:
|
||||
let a = pooled[variantName(v)]
|
||||
var ls, cs: string
|
||||
@@ -291,7 +349,14 @@ proc main() =
|
||||
ls.add &"{a.st.labHist[c]},"
|
||||
cs.add &"{a.st.choHist[c]},"
|
||||
echo &"\n# class histogram {variantName(v)}: labels=[{ls}] chosen=[{cs}] " &
|
||||
&"onlineAcc={a.st.classCorrect}/{a.st.classTotal}"
|
||||
&"onlineAcc={a.st.classCorrect}/{a.st.classTotal} " &
|
||||
&"radAcc={a.st.radCorrect}/{a.st.radTotal} " &
|
||||
&"revAcc={a.st.revCorrect}/{a.st.revTotal}"
|
||||
if variantName(v) in revLabTab:
|
||||
var rl, rdl: string
|
||||
for c in 0..<2: rl.add &"{revLabTab[variantName(v)][c]},"
|
||||
for c in 0..<TM_CLASSES: rdl.add &"{radLabTab[variantName(v)][c]},"
|
||||
echo &"# label hist {variantName(v)}: radial=[{rdl}] rev=[{rl}]"
|
||||
|
||||
# ── per-run distributions (a "run" = one fixture × one seed) ──
|
||||
# Linear is deterministic: replicate its one row per fixture across seeds so a
|
||||
@@ -313,6 +378,18 @@ proc main() =
|
||||
for s in 2..nSeeds:
|
||||
byVariant["Linear"][(row.fixture, s)] = byVariant["Linear"][(row.fixture, 1)]
|
||||
byVariantAll["Linear"][(row.fixture, s)] = byVariantAll["Linear"][(row.fixture, 1)]
|
||||
if vLinearOld in variants:
|
||||
for row in rows:
|
||||
if row.variant == "LinearOldClamp":
|
||||
for s in 2..nSeeds:
|
||||
byVariant["LinearOldClamp"][(row.fixture, s)] = byVariant["LinearOldClamp"][(row.fixture, 1)]
|
||||
byVariantAll["LinearOldClamp"][(row.fixture, s)] = byVariantAll["LinearOldClamp"][(row.fixture, 1)]
|
||||
if vTmpatBase in variants:
|
||||
for row in rows:
|
||||
if row.variant == "TMPatternBase":
|
||||
for s in 2..nSeeds:
|
||||
byVariant["TMPatternBase"][(row.fixture, s)] = byVariant["TMPatternBase"][(row.fixture, 1)]
|
||||
byVariantAll["TMPatternBase"][(row.fixture, s)] = byVariantAll["TMPatternBase"][(row.fixture, 1)]
|
||||
|
||||
echo "\n# ── per-run early-rate distribution (mean / min / max, n runs) ──"
|
||||
echo "variant,earlyMean%,earlyMin%,earlyMax%,overallMean%,overallMin%,overallMax%,n"
|
||||
|
||||
@@ -179,3 +179,188 @@ corrective excursion net-positive. Not directly measured.
|
||||
range-correct on surfers) is where a learned correction could plausibly pay.
|
||||
4. **Richer context.** 46% accuracy leaves room; the current context lacks the
|
||||
enemy's own recent GF history / segmentation that KNN/DecayGF exploit.
|
||||
|
||||
---
|
||||
|
||||
# ROUND 2 — fix the base, then try a target Linear cannot predict
|
||||
|
||||
Date: 2026-09-22. Artifacts: `common_libs/guns/tm_pattern.nim` (extended),
|
||||
`common_libs/tests/sweep_tm_pattern.nim` (extended). Raw outputs:
|
||||
`/tmp/tm2_real_path_s3.txt`, `/tmp/tm2_real_point_s3.txt`,
|
||||
`/tmp/tm_best_path_s3.txt`. All numbers below are MEASURED unless a line says
|
||||
INFERRED.
|
||||
|
||||
Round 1's "best" gun predicted the LATERAL GF bucket. Round 2 adds a RADIAL
|
||||
head (aim-distance correction) and a binary REVERSAL head (flip the GF sign),
|
||||
both on the same 40-bit context and the same TM core, selected by a runtime
|
||||
`targetMode` (`tmGF` | `tmRadial` | `tmReversal`). The shuffled-feedback control
|
||||
now randomises only the head the active mode is claiming.
|
||||
|
||||
## Task 1 — the base is EXACTLY Linear (premise refuted)
|
||||
|
||||
`tm_pattern`'s base is `forecastLinear`, which already iterates the flight time
|
||||
(5-iteration fixed point, same as `LinearGun`). The only deviation from
|
||||
`LinearGun` was the wall clamp: the base path clamped to `[BotRadius, W-BotRadius]`
|
||||
(17 px inset) instead of `LinearGun`'s `[0, W]`. Added a `forceBase` flag and a
|
||||
`TMPatternBase` variant, and made the zero-correction path return `f.x, f.y`
|
||||
with the exact `[0, W]` clamp.
|
||||
|
||||
Real DrussGT fixtures, bmPath, seeds=3, 18 fixture×seed runs, 77 rounds pooled:
|
||||
|
||||
| variant | early | overall |
|
||||
|---|---|---|
|
||||
| Linear | 34.0% (6358/18715) | 24.3% (58297/239943) |
|
||||
| LinearOldClamp (pre-fix base, BotRadius inset) | 34.2% (6350/18592) | 24.7% (59217/239891) |
|
||||
| **TMPatternBase (forceBase, exact Linear clamp)** | **34.0% (6358/18715)** | **24.3% (58297/239943)** |
|
||||
|
||||
Paired sign test Linear vs TMPatternBase: **18 ties, 0 wins each, p=1.000** on
|
||||
both early and overall; every per-run row (hits, shots, per-bin) is byte-for-byte
|
||||
identical. Per-run means identical: early 35.52%, overall 26.86%.
|
||||
|
||||
Verdict: **the base was never behind.** It IS `LinearGun` to the last floating
|
||||
point. The earlier "one-shot, non-iterating baseline" finding belonged to the OLD
|
||||
`guns/tsetlin.nim`, not to `tm_pattern`. The clamp fix is a wash (the old inset
|
||||
was marginally BETTER on overall: 24.7% vs 24.3%), so there is **zero baseline
|
||||
headroom** to recover: the entire deficit vs Linear is the TM's corrective
|
||||
excursions.
|
||||
|
||||
## Task 2 — radial target: a structural no-op under bmPath, a real WIN under bmPoint
|
||||
|
||||
**Structural fact (from `virtual_bullets.nim`, INFERRED then confirmed):** under
|
||||
`bmPath` a bullet flies along the aim RAY until it leaves the arena; the aim
|
||||
distance only sets `fireDist` (used for the tie-break probe), it does NOT change
|
||||
the ray. Moving the aim point radially along the base bearing therefore cannot
|
||||
change a `bmPath` hit. Confirmed exactly: on the synthetic set, `TMRadial` vs
|
||||
`Linear` scored **8/8 exact ties, p=1.000** under bmPath.
|
||||
|
||||
Under `bmPoint` the bullet resolves when `travelDist >= fireDist`, so the aim
|
||||
distance selects the arrival tick — the radial degree of freedom is live.
|
||||
|
||||
### bmPath (shipped), real, default config, seeds=3
|
||||
|
||||
| variant | early | overall |
|
||||
|---|---|---|
|
||||
| Linear | 34.0% | 24.3% |
|
||||
| TMRadial | 33.9% (19038/56203) | 24.1% (173577/719860) |
|
||||
| TMRadialShuf | 33.8% (18976/56085) | 24.3% (175154/719781) |
|
||||
|
||||
Per-run means (n=18): Linear 35.52/26.86; TMRadial 35.44/26.70; Shuf 35.40/26.92.
|
||||
Paired sign tests: Linear vs TMRadial early 13/5 p=0.096; overall 15/3 p=0.0075
|
||||
(a tiny systematic LOSS, traceable to the `BotRadius` clamp perturbing the ray
|
||||
near walls when `radOffset != 0`). TMRadial vs TMRadialShuf early 8/8 p=1.000.
|
||||
|
||||
**Verdict bmPath: no gain.** Radial is a structural no-op; the shipped metric
|
||||
therefore cannot reward Task 2.
|
||||
|
||||
### bmPoint, real, default config, seeds=3
|
||||
|
||||
| variant | early | overall |
|
||||
|---|---|---|
|
||||
| Linear | 7.2% (1480/20498) | 4.7% (11277/241423) |
|
||||
| Tsetlin (default gun) | 7.0% (4403/62617) | 4.8% (34588/724717) |
|
||||
| **TMRadial** | **9.4% (6013/63785)** | **5.8% (42079/726652)** |
|
||||
| TMRadialShuf (control) | 7.0% (4329/61691) | 3.6% (26116/724594) |
|
||||
|
||||
Per-run means (n=18): Linear 13.61/5.77; Tsetlin 13.13/5.82; TMRadial
|
||||
15.32/7.06; Shuf 12.86/4.34. Paired sign tests:
|
||||
|
||||
* **TMRadial > Linear: early 14/4 p=0.0309; overall 17/1 p=0.0001.**
|
||||
* **TMRadial > TMRadialShuf: early 17/1 p=0.0001; overall 18/0 p<0.0001.**
|
||||
* TMRadial > Tsetlin: early 17/1 p=0.0001; overall 15/3 p=0.0075.
|
||||
* TMRadialShuf vs Linear: early 10/8 p=0.81; overall 12/6 p=0.24 (control sits
|
||||
at baseline).
|
||||
|
||||
Online accuracy of the radial head: 48.8% (511949/1049453) vs **19.9%** shuffled
|
||||
chance under bmPoint (46.6% vs 20.0% under bmPath). Radial label histogram
|
||||
(raw, seeds=1) = [9058, 5392, 10305, 2236, 1112]: strongly asymmetric — surfers
|
||||
are often NEARER than the base constant-velocity prediction at the arrival tick
|
||||
(the base overshoots range on range-holders), so class 0 (aim 45–60 px short)
|
||||
dominates. That is the mechanism behind the win.
|
||||
|
||||
Caveat (MEASURED): `labelMiss` is much higher for radial mode (~4.3 M vs ~1.7 M
|
||||
for GF) because aiming SHORT resolves the bullet before the base arrival tick,
|
||||
so the arrival-tick ring sample is not yet recorded. The radial head is trained
|
||||
only on resolvable samples; the win is nonetheless measured on the metric, which
|
||||
is label-independent. A deferred-label fix would be the next refinement.
|
||||
|
||||
## Task 3 — binary reversal: not learnable, and the flip is a no-op
|
||||
|
||||
Label: net heading turn over the flight opposes the direction the enemy was
|
||||
turning at fire time (threshold 10°). Readout: train GF as in round 1, and if
|
||||
reversal is predicted, negate the GF correction (`TM_REV_GAIN=1.0`).
|
||||
|
||||
Label base rate (real, seeds=1, 1 round/fixture): `rev=[24772, 2673]` → the
|
||||
positive class is only **9.7%**. The head scores 86.8% (23030/26517) — **below
|
||||
the 90.3% majority-class base rate**, i.e. it is not detecting reversals at all,
|
||||
only predicting "no reversal". (The shuffled control is 50.2% because its labels
|
||||
are balanced.)
|
||||
|
||||
### bmPath (shipped), real, default config, seeds=3
|
||||
|
||||
| variant | early | overall |
|
||||
|---|---|---|
|
||||
| Linear | 34.0% | 24.3% |
|
||||
| Tsetlin (round-1 measurement) | 22.3% | 20.3% |
|
||||
| TMReversal | 19.5% (11062/56704) | 18.4% (132356/719725) |
|
||||
| TMReversalShuf | 19.1% (10925/57063) | 17.8% (128450/720126) |
|
||||
|
||||
Per-run means: TMReversal 23.45/20.32; Shuf 22.51/20.01. Paired: TMReversal vs
|
||||
Shuf early 12/6 p=0.238; overall 8/10 p=0.815 → **no learning effect on hits.**
|
||||
|
||||
### bmPath, best config (margin=0.25, shrink=0.5), real, seeds=3
|
||||
|
||||
| variant | early | overall |
|
||||
|---|---|---|
|
||||
| Linear | 34.0% | 24.3% |
|
||||
| TMPattern (gated GF, round-1 best) | 28.3% (15815/55902) | 22.2% (159674/719742) |
|
||||
| TMReversal (gated GF + flip) | 28.4% (15944/56067) | 22.3% (160163/719761) |
|
||||
| TMReversalShuf | 28.7% (16146/56195) | 21.7% (155973/719841) |
|
||||
|
||||
Paired: TMPattern vs TMReversal early 11/7 p=0.481, overall 7/11 p=0.481 — the
|
||||
flip changes nothing. TMReversal vs Shuf overall 13/5 p=0.096 (not significant).
|
||||
|
||||
**Verdict: clean negative.** The reversal target as defined is too rare to learn
|
||||
(head below the majority baseline), and using it to flip the GF sign is neutral
|
||||
to slightly negative on hits. Do not pursue this label; if revisited, balance the
|
||||
positive class (per-tick reversal events, or predict the arrival turn direction
|
||||
rather than "a reversal happened").
|
||||
|
||||
## Round-2 overall verdict
|
||||
|
||||
* On **bmPath (the shipped metric): the TM is NOT competitive with Linear.**
|
||||
Base = Linear exactly; radial is a structural no-op; gated GF is significantly
|
||||
worse (28.3%/22.2% vs 34.0%/24.3%, p=0.0075); reversal does nothing. The linear
|
||||
lead is already the best aim DIRECTION on these surfers and every learned
|
||||
angular excursion loses.
|
||||
* On **bmPoint: the TM now BEATS Linear and the default Tsetlin gun.**
|
||||
`TMRadial` (radial head, `TM_RADIAL_RANGE=60`, `TM_RAD_MARGIN=0.25`, 5 classes,
|
||||
gated), 9.4%/5.8% vs Linear 7.2%/4.7% (overall 17/1, p=0.0001) and vs Tsetlin
|
||||
7.0%/4.8% (overall 15/3, p=0.0075), with its shuffled control at 7.0%/3.6%.
|
||||
This is the first configuration in the whole TM effort that beats both
|
||||
baselines with a control-validated margin.
|
||||
* **Learning vs controls:** radial head 48.8% vs 19.9% chance (bmPoint); GF head
|
||||
reproduces round 1 (48.5% vs ~20%); reversal head does not beat majority.
|
||||
|
||||
Best configuration if the arrival-time metric is what matters: **TMRadial**. Best
|
||||
configuration under the shipped bmPath: **do nothing — keep the Linear base**. The
|
||||
evidence says the next step under bmPath is not another bucket target but either a
|
||||
richer DIRECTION representation (segmentation / pattern matching, as KNN and
|
||||
DecayGF use) or a metric that exposes the radial degree of freedom.
|
||||
|
||||
Per-enemy specialisation / freshness (MEASURED, unchanged from round 1): each
|
||||
offline round is replayed with a FRESH gun instance and the gun calls
|
||||
`resetLearning` if the target id changes mid-battle. The offline fixtures are
|
||||
single-target, so the mid-battle reset never fires there; its effect is untested
|
||||
by these numbers. There is no persistence across battles.
|
||||
|
||||
## MEASURED vs INFERRED (round 2)
|
||||
|
||||
* MEASURED: every table, per-run mean, paired sign test, online accuracy, label
|
||||
base rate, and the exact `TMPatternBase`/`Linear` byte-for-byte identity.
|
||||
* MEASURED: the bmPath radial no-op (synthetic exact ties; real bmPath tiny
|
||||
clamp-induced loss).
|
||||
* INFERRED: that bmPath ignores radial distance because it flies a ray — read
|
||||
from `virtual_bullets.nim`, then confirmed by the synthetic tie.
|
||||
* INFERRED: that the radial win comes from surfers being NEARER than the base
|
||||
prediction (range-holding), supported by the asymmetric radial label histogram
|
||||
but not separately modelled.
|
||||
|
||||
Reference in New Issue
Block a user