SirStone
be74e369eb
problem gives 60%. Thread closed with a number, not a shrug.
Gate 2 (fd2f7f6) said STOP because every correction lost hits. But the shifts it
applied were the conditional MEDIANS of the error (4-16 deg) while the gate metric
is HITS, and the naive guess is UNBIASED - so shifting an already-centred
distribution can only destroy near-target mass. That suggested the experiment was
MISCALIBRATED rather than the idea being dead. One cheap test settled it.
ARMS (pooled, both primary fixtures, 25 rounds, N~9433/h; deltas in pp vs naive):
h naive TM-med TM-hit TM-x0.25 PERF-SIGN shuffled
15 39.3 29.0(-10.3) 35.1(-4.2) 37.9(-1.5) **47.0 (+7.7)** 35.1(-4.2)
20 28.2 19.2(-9.0) 22.8(-5.4) 25.6(-2.6) **33.4 (+5.1)** 22.0(-6.2)
25 20.9 13.1(-7.8) 15.4(-5.5) 18.8(-2.2) **25.6 (+4.7)** 16.0(-4.9)
30 16.4 10.5(-5.9) 11.4(-5.0) 14.1(-2.3) **18.3 (+1.9)** 11.2(-5.2)
1. **THE FORM CAN BUY HITS.** Perfect side knowledge + the hit-optimal shift gains
**+4.9 pp mean** (+1.9..+7.7), positive in ~19/25 rounds and in BOTH fixtures -
and it improves the median AND p90 residual. So the premise is NOT structurally
impossible.
2. **BUT THE TM'S 60% SIDE ACCURACY LOSES** (-4.2 pp quadrant, -3.3..-5.6 side).
Shrinking the shift toward zero monotonically reduces the loss but never turns
it positive.
3. **BREAK-EVEN IS ~80% SIDE ACCURACY** (synthetic sweep: 0.70 -> -1.6 pp, **0.80
-> 0.0**, 1.00 -> +4.9 pp). The observed side signal tops out at **~60% (TM) /
53-56% (turn-only rule)** against 50% chance, and the headroom study found it is
essentially ONE WEAK FEATURE. **No realistic predictor of this problem clears
the 80% wall.**
WHY IT IS SO SENSITIVE - the hits-vs-shift curve (h15, calibration): shift 0 ->
36.2%, +-2deg -> 27.4/26.8, +-4deg -> 14.2, +-6deg -> 11.7, +-10deg -> 8.3.
Shifting unconditionally is catastrophic. The whole value comes from shifting ONLY
when the side is known, because the signed-error distribution becomes ONE-SIDED
once you condition on the true side. And the hit-optimal PERF-SIGN shift is only
**+-2 to 3.5 deg** - an order of magnitude smaller than the 4-16 deg medians Gate 2
used, which is exactly why Gate 2's calibration could never win.
VERDICT: **MISCALIBRATED, but practically dead at achievable accuracy.**
Recommendation: stop the per-bucket-shift direction, and record the RIGHT reason -
an **accuracy wall at ~80%**, not an impossibility of the form. If ever revived,
the only viable path is a side predictor that materially exceeds 60%; a better
calibration cannot fix it (we already used the hit-optimal one).
METHOD NOTE: the hit-optimal shift is fitted on the calibration slice (the
pipeline's existing out-of-sample offset split) and applied on the later eval
slice - NOT fitted on the TM training slice, to avoid in-sample label leakage.
INTEGRITY: TM-med reproduces Gate 2's committed deltas to the decimal
(-10.3/-9.0/-7.8/-5.9); PS-shift0 and TM-x0.0 are exactly naive; the shuffled
control never improves. Guards all pass (test_tm_diag 48, test_tm_automata_diag 55,
test_tm_clause_shape 66, diag_synthetic, diag_automata_validation, test_gun_harness,
test_vbullet_metric, test_power_selection, test_adaptive_radar,
test_tfil_ring_weights, test_power_policy, test_ram_decision, test_rack_membership,
test_selector_tiebreak, test_tm_pattern_registration, test_vbullet_admit_gate).
Caveats: DrussGT-only; enemy movement is a closed-loop response to our CURRENT
movement; offline observation is perfect while live we see the enemy only on scans
-> all absolute hit fractions are optimistic upper bounds, only deltas are
meaningful.
1055 lines
40 KiB
Nim
1055 lines
40 KiB
Nim
## GATE 2b — is the "shift the aim per predicted bucket" application form
|
|
## STRUCTURALLY DEAD, or merely MISCALIBRATED? OFFLINE ONLY.
|
|
##
|
|
## Gate 2 (fd2f7f6) calibrated each quadrant to the conditional MEDIAN error and
|
|
## LOST hits everywhere. The gate metric is HITS, not median error, so the
|
|
## calibration objective was wrong. This file changes ONLY the calibration /
|
|
## appliance step: each bucket's offset is the shift (fine grid, -20..+20 deg in
|
|
## 0.5 deg steps) that MAXIMISES THE HIT COUNT on the out-of-sample calibration
|
|
## slice. It adds the DECISIVE arm: PERFECT-SIGN (cheating, the true sign) with
|
|
## the hit-optimal shift. If even perfect side knowledge cannot beat naive on
|
|
## hits, the application form is structurally dead.
|
|
##
|
|
## Original Gate 2 pipeline (unchanged): extract the draft 49-bit spec + 4-bit
|
|
## horizon, derive quadrant labels, train tm_core fresh per round, split each
|
|
## round train|calibrate|eval, measure residual angular error + hit fraction.
|
|
##
|
|
## Pipeline (all offline, fixtures READ-ONLY):
|
|
## 1. Extract the DRAFT 49-bit TM feature spec (walls / us / motion / bullets)
|
|
## from the committed DrussGT fixtures, plus a 4-bit one-hot horizon block
|
|
## (h in {15,20,25,30}) -> 53 raw bits.
|
|
## 2. Build FACT labels: for a sample at tick t and horizon h, look up where
|
|
## the enemy ACTUALLY was at t+h (never across a round boundary; the last h
|
|
## ticks of each round are dropped). Two binaries:
|
|
## (a) side: enemy LEFT / RIGHT of the naive straight-line guess,
|
|
## (b) magnitude: |angular error| bigger / smaller than the TRAIN median.
|
|
## Four quadrants: left-small / left-big / right-small / right-big.
|
|
## 3. Train a real Tsetlin Machine with the validated core at
|
|
## `common_libs/tm_diag/tm_core.nim` (one fresh model per round = the
|
|
## intended "fresh every round, overfit the current enemy" semantics).
|
|
## 4. Map each predicted quadrant to a representative signed angular offset
|
|
## (median signed error of the TRAINING samples in that quadrant), apply it
|
|
## to the naive aim, and measure the residual angular error.
|
|
##
|
|
## Four arms + floor:
|
|
## naive : straight-line guess (baseline)
|
|
## TM : trained model
|
|
## shuffled : same pipeline, labels randomised (pipeline-integrity control)
|
|
## turn-only : uses ONLY the enemy's current turn direction (critical arm)
|
|
## majority : constant majority-quadrant offset (floor)
|
|
##
|
|
## Metric: median / p90 residual |angular error| (deg) and the estimated hit
|
|
## fraction (|residual| < atan(18px / range)). The absolute hit fraction is
|
|
## OPTIMISTIC (perfect arrival knowledge every tick = bmPoint-style); only the
|
|
## DELTA before/after is meaningful.
|
|
##
|
|
## Protocol: within-round split, train on the EARLY portion, evaluate on the
|
|
## LATER portion. Focus horizons h = 15,20,25 (+30).
|
|
##
|
|
## Run: nim c -r -d:release --path:common_libs \
|
|
## common_libs/tests/measure_tm_miss_shrink.nim
|
|
## [--fixtures=a,b] [--epochs=N] [--trainfrac=F] [--clauses=N]
|
|
|
|
import std/[json, os, strformat, strutils, math, algorithm, tables]
|
|
import tm_diag/tm_core
|
|
|
|
# ── configuration ────────────────────────────────────────────────────────────
|
|
|
|
const repoRoot* = currentSourcePath().parentDir.parentDir.parentDir
|
|
const fixturesDir* = repoRoot / "tools" / "fixtures"
|
|
const metaDir* = fixturesDir / "drussgt_meta"
|
|
|
|
const HORIZONS = [15, 20, 25, 30]
|
|
const NH = 4
|
|
const N_BASE = 49 # draftTMSpec() bit count
|
|
const N_BITS = N_BASE + 4 # + 4-bit horizon one-hot
|
|
const N_CLASSES = 4 # left-small, left-big, right-small, right-big
|
|
const MIN_I = 12 # need 10 ticks of history for the motion features
|
|
const BOT_RADIUS = 18.0
|
|
|
|
## Hit-optimal shift search grid: -20..+20 deg in 0.5 deg steps.
|
|
const SHIFT_LO = -20.0
|
|
const SHIFT_HI = 20.0
|
|
const SHIFT_STEP = 0.5
|
|
const NSHIFT = int((SHIFT_HI - SHIFT_LO) / SHIFT_STEP) + 1 # 81
|
|
|
|
## Hits-vs-shift-magnitude diagnostic curve: -20..+20 deg in 2 deg steps.
|
|
const NCURVE = 21
|
|
proc curveShift(k: int): float {.inline.} = -20.0 + 2.0 * float(k)
|
|
|
|
## Side-accuracy sweep: a synthetic side predictor that is correct with
|
|
## probability q (independent draws on calibration and eval).
|
|
const NQ = 7
|
|
const ACCS = [0.50, 0.55, 0.60, 0.65, 0.70, 0.80, 1.00]
|
|
|
|
proc shiftGrid(): seq[float] =
|
|
for k in 0..<NSHIFT: result.add SHIFT_LO + SHIFT_STEP * float(k)
|
|
|
|
var
|
|
cfgClauses = 50
|
|
cfgStates = 64
|
|
cfgS = 3.0
|
|
cfgEpochs = 5
|
|
cfgTrainFrac = 0.70
|
|
|
|
const PRIMARY = ["tr_drussgt_vs_modularbot.jsonl",
|
|
"tr_drussgt_vs_modularbot_shield.jsonl"]
|
|
|
|
# ── small helpers ────────────────────────────────────────────────────────────
|
|
|
|
proc wrap180(a: float): float {.inline.} =
|
|
var r = a
|
|
while r > 180.0: r -= 360.0
|
|
while r <= -180.0: r += 360.0
|
|
r
|
|
|
|
proc signf(x: float): int {.inline.} =
|
|
if x > 1e-9: 1 elif x < -1e-9: -1 else: 0
|
|
|
|
proc jf(d: JsonNode, k: string): float =
|
|
let n = d[k]
|
|
case n.kind
|
|
of JFloat: n.getFloat
|
|
of JInt: float(n.getInt)
|
|
else: parseFloat(n.getStr)
|
|
|
|
# ── data model ───────────────────────────────────────────────────────────────
|
|
|
|
type
|
|
Tick* = object
|
|
tick*: int
|
|
ex*, ey*, eh*, es*, ee*: float
|
|
sx*, sy*, sh*, ss*, se*: float
|
|
|
|
Rnd* = object
|
|
roundNo*: int
|
|
st*: seq[Tick]
|
|
|
|
BulletSeries* = object
|
|
## Per-tick proxy for OUR in-flight bullets (INFERRED from self-energy
|
|
## drops; the fixture records no gun heading/power).
|
|
tta*: seq[int] ## ticks until the nearest in-flight bullet arrives, -1 none
|
|
lat*: seq[float] ## lateral offset of the enemy from the fired path (px)
|
|
|
|
## Arms:
|
|
## aNaive : straight-line, no shift (baseline)
|
|
## aTMmed : OLD gate-2 appliance (conditional median per predicted quadrant)
|
|
## aTMhit : TM quadrant -> HIT-OPTIMAL shift (shrink x1.0)
|
|
## aTMside : TM predicted SIDE (quadrant collapsed) -> hit-optimal shift
|
|
## aTMs0 : TM bucket -> hit-optimal shift x0.0 (== naive, sanity)
|
|
## aTMs25 : TM bucket -> hit-optimal shift x0.25
|
|
## aTMs50 : TM bucket -> hit-optimal shift x0.50
|
|
## aPS : PERFECT-SIGN -> hit-optimal shift (DECISIVE, cheating)
|
|
## aPS0 : PERFECT-SIGN, shift 0 (== naive, sanity)
|
|
## aShuf : shuffled-label model -> hit-optimal shift (integrity control)
|
|
Arm = enum aNaive, aTMmed, aTMhit, aTMside, aTMs0, aTMs25, aTMs50,
|
|
aPS, aPS0, aShuf
|
|
Hist = object
|
|
errs: array[Arm, array[NH, seq[float]]]
|
|
hits: array[Arm, array[NH, int]]
|
|
n: array[NH, int]
|
|
# diagnostics
|
|
tmQCor: array[NH, int] ## TM predicted the true quadrant
|
|
tmQTot: array[NH, int]
|
|
tmHitCor: array[NH, int] ## TM predicted the true sign
|
|
tmHitTot: array[NH, int]
|
|
shufQCor: array[NH, int] ## shuffled-model predicted the true quadrant
|
|
shufQTot: array[NH, int]
|
|
predCounts: array[NH, array[N_CLASSES, int]]
|
|
trueCounts: array[NH, array[N_CLASSES, int]]
|
|
# fitted hit-optimal shifts (diagnostic)
|
|
fitTM: array[NH, array[N_CLASSES, float]]
|
|
fitTMSide: array[NH, array[2, float]]
|
|
fitShuf: array[NH, array[N_CLASSES, float]]
|
|
fitPS: array[NH, array[2, float]]
|
|
# hits-vs-constant-shift curve, pooled calibration samples per horizon
|
|
curveHits: array[NH, array[NCURVE, int]]
|
|
curveN: array[NH, int]
|
|
# side-accuracy sweep: hits on eval for a synthetic q-accurate predictor
|
|
accHits: array[NQ, array[NH, int]]
|
|
accN: array[NQ, array[NH, int]]
|
|
accFit: array[NQ, array[NH, array[2, float]]]
|
|
|
|
# ── fixture loading ──────────────────────────────────────────────────────────
|
|
|
|
proc loadTicks(path: string): seq[Tick] =
|
|
for line in lines(path):
|
|
let ln = line.strip()
|
|
if ln.len == 0: continue
|
|
let d = parseJson(ln)
|
|
if not d.hasKey("tick"): continue
|
|
result.add Tick(tick: d["tick"].getInt,
|
|
ex: jf(d, "ex"), ey: jf(d, "ey"), eh: jf(d, "eh"),
|
|
es: jf(d, "es"), ee: jf(d, "ee"),
|
|
sx: jf(d, "sx"), sy: jf(d, "sy"), sh: jf(d, "sh"),
|
|
ss: jf(d, "ss"), se: jf(d, "se"))
|
|
|
|
proc loadRounds(path: string, ticks: seq[Tick]): seq[Rnd] =
|
|
let rp = metaDir / (extractFilename(path) & ".rounds.json")
|
|
var spans: seq[(int, int)]
|
|
if fileExists(rp):
|
|
let j = parseFile(rp)
|
|
for r in j["rounds"]:
|
|
spans.add (r["startTick"].getInt, r["count"].getInt)
|
|
elif ticks.len > 0:
|
|
spans.add (ticks[0].tick, ticks.len)
|
|
var idxByTick = initTable[int, int]()
|
|
for i, t in ticks: idxByTick[t.tick] = i
|
|
for sp in spans:
|
|
let (s0, c) = sp
|
|
if not idxByTick.hasKey(s0): continue
|
|
let i0 = idxByTick[s0]
|
|
var st: seq[Tick]
|
|
for k in 0..<c:
|
|
if i0 + k < ticks.len: st.add ticks[i0 + k]
|
|
if st.len > 0:
|
|
result.add Rnd(roundNo: result.len + 1, st: st)
|
|
|
|
# ── bullet proxy (INFERRED) ──────────────────────────────────────────────────
|
|
|
|
proc buildBulletSeries(r: Rnd): BulletSeries =
|
|
let L = r.st.len
|
|
result.tta = newSeq[int](L)
|
|
for i in 0..<L: result.tta[i] = -1
|
|
result.lat = newSeq[float](L)
|
|
for t0 in 1..<L:
|
|
let drop = r.st[t0 - 1].se - r.st[t0].se
|
|
if drop <= 0.05 or drop > 3.1: continue # damage, not a fire
|
|
var power = drop
|
|
if power < 0.1: power = 0.1
|
|
if power > 3.0: power = 3.0
|
|
let speed = 20.0 - 3.0 * power
|
|
let rng = hypot(r.st[t0].ex - r.st[t0].sx, r.st[t0].ey - r.st[t0].sy)
|
|
let flight = int(ceil(rng / speed))
|
|
let dx = r.st[t0].ex - r.st[t0].sx
|
|
let dy = r.st[t0].ey - r.st[t0].sy
|
|
let nrm = max(1e-6, hypot(dx, dy))
|
|
let ux = dx / nrm
|
|
let uy = dy / nrm
|
|
for k in 0..flight:
|
|
let t = t0 + k
|
|
if t >= L: break
|
|
let ta = t0 + flight - t
|
|
if result.tta[t] < 0 or ta < result.tta[t]:
|
|
result.tta[t] = ta
|
|
let vx = r.st[t].ex - r.st[t0].sx
|
|
let vy = r.st[t].ey - r.st[t0].sy
|
|
result.lat[t] = ux * vy - uy * vx
|
|
|
|
# ── feature extraction: the 49 draft bits (causal, at tick i) ────────────────
|
|
#
|
|
# Block layout (mirrors draftTMSpec()):
|
|
# 0..3 dist-to-nearest-wall (4 one-hot)
|
|
# 4..7 which-wall-nearest (4 one-hot)
|
|
# 8..13 dist-from-us (6 one-hot)
|
|
# 14..16 enemy-heading-vs-line-to-us (3 one-hot)
|
|
# 17..19 turn-direction t, t-1, t-2 (3 boolean "was turning left")
|
|
# 20..24 ticks-since-reversal (5 one-hot)
|
|
# 25..27 turn-consistency-10 (3 one-hot)
|
|
# 28..30 distance-moved-10 (3 one-hot)
|
|
# 31..33 speed-trend-10 (3 one-hot)
|
|
# 34..36 turn-rate-change-5 (3 one-hot)
|
|
# 37..41 time-until-bullet (5 one-hot)
|
|
# 42..48 bullet-lateral-offset (7 one-hot)
|
|
|
|
proc buildBase(r: Rnd, bi: BulletSeries, i: int, sinceRev: seq[int]): array[N_BASE, int] =
|
|
let s = r.st
|
|
let cur = s[i]
|
|
|
|
# walls
|
|
let dL = cur.ex
|
|
let dR = 800.0 - cur.ex
|
|
let dT = 600.0 - cur.ey
|
|
let dBottom = cur.ey
|
|
let dmin = min(min(dL, dR), min(dT, dBottom))
|
|
var wallBin = 3
|
|
if dmin < 50.0: wallBin = 0
|
|
elif dmin < 100.0: wallBin = 1
|
|
elif dmin < 200.0: wallBin = 2
|
|
result[wallBin] = 1
|
|
var wb = 0
|
|
let walls = [dL, dR, dT, dBottom]
|
|
for w in 1..3:
|
|
if walls[w] < walls[wb]: wb = w
|
|
result[4 + wb] = 1
|
|
|
|
# us
|
|
let rng = hypot(cur.ex - cur.sx, cur.ey - cur.sy)
|
|
var ub = 5
|
|
if rng < 100.0: ub = 0
|
|
elif rng < 200.0: ub = 1
|
|
elif rng < 300.0: ub = 2
|
|
elif rng < 400.0: ub = 3
|
|
elif rng < 600.0: ub = 4
|
|
result[8 + ub] = 1
|
|
|
|
let lane = arctan2(cur.sy - cur.ey, cur.sx - cur.ex)
|
|
let hdg = cur.eh * PI / 180.0
|
|
let perp = abs(sin(hdg - lane))
|
|
var hb = 1
|
|
if perp < 0.5: hb = 2
|
|
elif perp > 0.866: hb = 0
|
|
result[14 + hb] = 1
|
|
|
|
# motion: turn direction
|
|
for k in 0..2:
|
|
if i - 1 - k >= 0:
|
|
let d = wrap180(s[i - k].eh - s[i - 1 - k].eh)
|
|
if d > 1e-6: result[17 + k] = 1
|
|
|
|
# ticks since reversal
|
|
var rb = 4
|
|
let sr = sinceRev[i]
|
|
if sr < 5: rb = 0
|
|
elif sr < 10: rb = 1
|
|
elif sr < 20: rb = 2
|
|
elif sr < 40: rb = 3
|
|
result[20 + rb] = 1
|
|
|
|
# turn consistency over last 10
|
|
var pos = 0
|
|
var neg = 0
|
|
for k in 0..9:
|
|
if i - 1 - k < 0: break
|
|
let d = wrap180(s[i - k].eh - s[i - 1 - k].eh)
|
|
if d > 1e-6: inc pos
|
|
elif d < -1e-6: inc neg
|
|
let tot = pos + neg
|
|
let cons = if tot > 0: max(pos, neg).float / tot.float else: 0.0
|
|
var cb = 0
|
|
if cons > 0.8: cb = 2
|
|
elif cons >= 0.5: cb = 1
|
|
result[25 + cb] = 1
|
|
|
|
# distance moved over 10
|
|
let j0 = max(0, i - 10)
|
|
let dm = hypot(cur.ex - s[j0].ex, cur.ey - s[j0].ey)
|
|
var mb = 1
|
|
if dm < 20.0: mb = 0
|
|
elif dm > 50.0: mb = 2
|
|
result[28 + mb] = 1
|
|
|
|
# speed trend over 10
|
|
let st10 = abs(s[max(0, i - 10)].es)
|
|
let spdDiff = abs(cur.es) - st10
|
|
var sb = 1
|
|
if spdDiff < -0.5: sb = 0
|
|
elif spdDiff > 0.5: sb = 2
|
|
result[31 + sb] = 1
|
|
|
|
# turn-rate change: last 5 deltas vs previous 5
|
|
var r1 = 0.0
|
|
var n1 = 0
|
|
for k in 0..4:
|
|
if i - 1 - k >= 0:
|
|
r1 += abs(wrap180(s[i - k].eh - s[i - 1 - k].eh)); inc n1
|
|
var r2 = 0.0
|
|
var n2 = 0
|
|
for k in 5..9:
|
|
if i - 1 - k >= 0:
|
|
r2 += abs(wrap180(s[i - k].eh - s[i - 1 - k].eh)); inc n2
|
|
let m1 = if n1 > 0: r1 / float(n1) else: 0.0
|
|
let m2 = if n2 > 0: r2 / float(n2) else: 0.0
|
|
let dtr = m1 - m2
|
|
var tb = 1
|
|
if dtr < -0.3: tb = 0
|
|
elif dtr > 0.3: tb = 2
|
|
result[34 + tb] = 1
|
|
|
|
# bullets
|
|
let tta = bi.tta[i]
|
|
var b1 = 0
|
|
if tta >= 0:
|
|
if tta < 5: b1 = 1
|
|
elif tta < 10: b1 = 2
|
|
elif tta < 20: b1 = 3
|
|
else: b1 = 4
|
|
result[37 + b1] = 1
|
|
|
|
let lat = bi.lat[i]
|
|
var lb = 3
|
|
if lat < -72.0: lb = 0
|
|
elif lat < -36.0: lb = 1
|
|
elif lat < -18.0: lb = 2
|
|
elif lat <= 18.0: lb = 3
|
|
elif lat <= 36.0: lb = 4
|
|
elif lat <= 72.0: lb = 5
|
|
else: lb = 6
|
|
result[42 + lb] = 1
|
|
|
|
proc toLits(base: array[N_BASE, int], h: int): seq[uint8] =
|
|
var raw: array[N_BITS, int]
|
|
for i in 0..<N_BASE: raw[i] = base[i]
|
|
for k, hh in HORIZONS:
|
|
if hh == h: raw[N_BASE + k] = 1
|
|
result = newSeq[uint8](2 * N_BITS)
|
|
for i in 0..<N_BITS:
|
|
let v = uint8(if raw[i] != 0: 1 else: 0)
|
|
result[i] = v
|
|
result[i + N_BITS] = 1'u8 - v
|
|
|
|
proc sinceRevSeries(r: Rnd): seq[int] =
|
|
let L = r.st.len
|
|
result = newSeq[int](L)
|
|
var lastFlip = -1
|
|
var prevSg = 0
|
|
for i in 0..<L:
|
|
let sg = signf(r.st[i].es)
|
|
if sg != 0:
|
|
if prevSg != 0 and sg != prevSg: lastFlip = i
|
|
prevSg = sg
|
|
result[i] = if lastFlip < 0: i + 1000 else: i - lastFlip
|
|
|
|
# ── TM training (reuses tm_core; no per-sample allocation) ───────────────────
|
|
|
|
proc trainEpoch(m: var TmMachine, lits: seq[seq[uint8]], labels: seq[int],
|
|
caches: var seq[seq[uint8]], votes: var seq[float],
|
|
order: seq[int]) =
|
|
for s in order:
|
|
for c in 0..<m.nClasses:
|
|
votes[c] = tmForward(m, m.teams[c], lits[s], caches[c])
|
|
for c in 0..<m.nClasses:
|
|
let d = if c == labels[s]: 1.0 else: -1.0
|
|
tmLearnDir(m, m.teams[c], lits[s], caches[c], votes[c], d)
|
|
|
|
proc trainMachine(m: var TmMachine, lits: seq[seq[uint8]], labels: seq[int],
|
|
epochs: int, seed: uint64) =
|
|
m.resetMachine(seed)
|
|
var caches = newSeq[seq[uint8]](m.nClasses)
|
|
for c in 0..<m.nClasses:
|
|
caches[c] = newSeq[uint8](m.nClauses)
|
|
var votes = newSeq[float](m.nClasses)
|
|
var order = newSeq[int](lits.len)
|
|
for i in 0..<lits.len: order[i] = i
|
|
for _ in 0..<epochs:
|
|
for i in countdown(order.len - 1, 1):
|
|
let j = int(m.rng.nextU64() mod uint64(i + 1))
|
|
swap(order[i], order[j])
|
|
trainEpoch(m, lits, labels, caches, votes, order)
|
|
|
|
proc predictWith(m: TmMachine, lits: openArray[uint8],
|
|
cache: var seq[uint8]): int =
|
|
var bestV = -Inf
|
|
for c in 0..<m.nClasses:
|
|
let v = tmForward(m, m.teams[c], lits, cache)
|
|
if v > bestV:
|
|
bestV = v
|
|
result = c
|
|
|
|
# ── statistics ───────────────────────────────────────────────────────────────
|
|
|
|
proc medOf(v: seq[float]): float =
|
|
if v.len == 0: return NaN
|
|
var s = v
|
|
s.sort()
|
|
s[s.len div 2]
|
|
|
|
proc qOf(v: seq[float], q: float): float =
|
|
if v.len == 0: return NaN
|
|
var s = v
|
|
s.sort()
|
|
s[min(s.len - 1, max(0, int(q * float(s.len - 1) + 0.5)))]
|
|
|
|
proc medOfInts(v: seq[int]): float =
|
|
if v.len == 0: return NaN
|
|
var s = v
|
|
s.sort()
|
|
float(s[s.len div 2])
|
|
|
|
proc fitHitOptimal(errs, halves: seq[float], grid: seq[float]): float =
|
|
## The shift on `grid` that MAXIMISES the number of hits
|
|
## (|err - shift| < half) on this bucket. Ties prefer the
|
|
## smallest-magnitude shift, so an optimum of 0 is reported as 0.
|
|
var best = 0.0
|
|
var bestHits = -1
|
|
for sh in grid:
|
|
var hits = 0
|
|
for k in 0..<errs.len:
|
|
if abs(errs[k] - sh) < halves[k]: inc hits
|
|
if hits > bestHits or (hits == bestHits and abs(sh) < abs(best)):
|
|
bestHits = hits
|
|
best = sh
|
|
best
|
|
|
|
proc mergeHist(dst: var Hist, src: Hist) =
|
|
for a in Arm:
|
|
for hi in 0..<NH:
|
|
dst.errs[a][hi].add src.errs[a][hi]
|
|
dst.hits[a][hi] += src.hits[a][hi]
|
|
for hi in 0..<NH:
|
|
dst.n[hi] += src.n[hi]
|
|
dst.tmQCor[hi] += src.tmQCor[hi]
|
|
dst.tmQTot[hi] += src.tmQTot[hi]
|
|
dst.tmHitCor[hi] += src.tmHitCor[hi]
|
|
dst.tmHitTot[hi] += src.tmHitTot[hi]
|
|
dst.shufQCor[hi] += src.shufQCor[hi]
|
|
dst.shufQTot[hi] += src.shufQTot[hi]
|
|
for c in 0..<N_CLASSES:
|
|
dst.predCounts[hi][c] += src.predCounts[hi][c]
|
|
dst.trueCounts[hi][c] += src.trueCounts[hi][c]
|
|
for c in 0..<N_CLASSES:
|
|
dst.fitTM[hi][c] = src.fitTM[hi][c]
|
|
dst.fitShuf[hi][c] = src.fitShuf[hi][c]
|
|
for sd in 0..1:
|
|
dst.fitPS[hi][sd] = src.fitPS[hi][sd]
|
|
dst.fitTMSide[hi][sd] = src.fitTMSide[hi][sd]
|
|
for k in 0..<NCURVE: dst.curveHits[hi][k] += src.curveHits[hi][k]
|
|
dst.curveN[hi] += src.curveN[hi]
|
|
for qi in 0..<NQ:
|
|
dst.accHits[qi][hi] += src.accHits[qi][hi]
|
|
dst.accN[qi][hi] += src.accN[qi][hi]
|
|
for sd in 0..1: dst.accFit[qi][hi][sd] = src.accFit[qi][hi][sd]
|
|
|
|
const ArmName: array[Arm, string] =
|
|
["naive", "TM-med", "TM-hit", "TM-side", "TM-x0.0", "TM-x0.25", "TM-x0.5",
|
|
"PERF-SIGN", "PS-shift0", "shuffled"]
|
|
|
|
## Arms shown in the per-round table (pooled table shows all).
|
|
const ShowArms = [aNaive, aTMmed, aTMhit, aTMside, aTMs25, aTMs50, aPS, aShuf]
|
|
|
|
# ── per-round pipeline ───────────────────────────────────────────────────────
|
|
|
|
proc runRound(r: Rnd, bi: BulletSeries, tmCache: var seq[uint8],
|
|
emit: bool): Hist =
|
|
let s = r.st
|
|
let L = s.len
|
|
var base = newSeq[array[N_BASE, int]](L)
|
|
block:
|
|
let sr = sinceRevSeries(r)
|
|
for i in 0..<L: base[i] = buildBase(r, bi, i, sr)
|
|
|
|
# Three-way WITHIN-ROUND split: fit | calibrate | eval. The model is never
|
|
# trained on the ticks it is evaluated on, and the quadrant->offset mapping is
|
|
# calibrated OUT-OF-SAMPLE so an overfit in-sample median cannot leak.
|
|
let fitEnd = max(MIN_I + 1, int(0.50 * float(L)))
|
|
let calEnd = max(fitEnd + 1, int(cfgTrainFrac * float(L)))
|
|
|
|
# local collector: samples with i in [lo,stop) and the label j = i+h < stop
|
|
# (so NOTHING here reads past `stop` — no cross-region label leakage).
|
|
proc collect(lo, stop, hidx: int):
|
|
tuple[hs: seq[int], es: seq[float], ls: seq[seq[uint8]],
|
|
ts: seq[float], rs: seq[float]] =
|
|
let h = HORIZONS[hidx]
|
|
for i in lo..<stop:
|
|
let j = i + h
|
|
if j >= stop: continue
|
|
let cur = s[i]
|
|
let gx = cur.ex + cur.es * cos(cur.eh * PI / 180.0) * float(h)
|
|
let gy = cur.ey + cur.es * sin(cur.eh * PI / 180.0) * float(h)
|
|
let ba = arctan2(s[j].ey - cur.sy, s[j].ex - cur.sx)
|
|
let bg = arctan2(gy - cur.sy, gx - cur.sx)
|
|
let err = radToDeg(arctan2(sin(ba - bg), cos(ba - bg)))
|
|
result.hs.add hidx
|
|
result.es.add err
|
|
result.ls.add toLits(base[i], h)
|
|
result.ts.add(if i - 1 >= 0: wrap180(cur.eh - s[i - 1].eh) else: 0.0)
|
|
result.rs.add hypot(s[j].ex - cur.sx, s[j].ey - cur.sy)
|
|
|
|
var trH: seq[int]
|
|
var trErr: seq[float]
|
|
var trLits: seq[seq[uint8]]
|
|
var trTurn: seq[float]
|
|
var calH: seq[int]
|
|
var calErr: seq[float]
|
|
var calLits: seq[seq[uint8]]
|
|
var calTurn: seq[float]
|
|
var calRange: seq[float]
|
|
var evH: seq[int]
|
|
var evIdx: seq[int]
|
|
var evErr: seq[float]
|
|
var evRange: seq[float]
|
|
var evTurn: seq[float]
|
|
|
|
for hi in 0..<NH:
|
|
let h = HORIZONS[hi]
|
|
let a = collect(MIN_I, fitEnd, hi)
|
|
trH.add a.hs; trErr.add a.es; trLits.add a.ls; trTurn.add a.ts
|
|
let b = collect(fitEnd, calEnd, hi)
|
|
calH.add b.hs; calErr.add b.es; calLits.add b.ls; calTurn.add b.ts
|
|
calRange.add b.rs
|
|
for i in calEnd..<L:
|
|
let j = i + h
|
|
if j >= L: continue
|
|
let cur = s[i]
|
|
let gx = cur.ex + cur.es * cos(cur.eh * PI / 180.0) * float(h)
|
|
let gy = cur.ey + cur.es * sin(cur.eh * PI / 180.0) * float(h)
|
|
let ba = arctan2(s[j].ey - cur.sy, s[j].ex - cur.sx)
|
|
let bg = arctan2(gy - cur.sy, gx - cur.sx)
|
|
let err = radToDeg(arctan2(sin(ba - bg), cos(ba - bg)))
|
|
evH.add hi
|
|
evIdx.add i
|
|
evErr.add err
|
|
evRange.add hypot(s[j].ex - cur.sx, s[j].ey - cur.sy)
|
|
evTurn.add(if i - 1 >= 0: wrap180(cur.eh - s[i - 1].eh) else: 0.0)
|
|
|
|
# ── labels (true quadrants); magnitude threshold = FIT median |err| ──
|
|
var medAbs: array[NH, float]
|
|
for hi in 0..<NH:
|
|
var absE: seq[float]
|
|
for k in 0..<trErr.len:
|
|
if trH[k] == hi: absE.add abs(trErr[k])
|
|
medAbs[hi] = medOf(absE)
|
|
|
|
# assemble FIT training set (drop exactly-zero error ticks — no side)
|
|
var tlits: seq[seq[uint8]]
|
|
var terr: seq[float]
|
|
var th: seq[int]
|
|
var tturn: seq[float]
|
|
var tlab: seq[int]
|
|
for k in 0..<trErr.len:
|
|
if abs(trErr[k]) < 1e-9: continue
|
|
let hi = trH[k]
|
|
let cls = ((if trErr[k] > 0.0: 0 else: 2) +
|
|
(if abs(trErr[k]) > medAbs[hi]: 1 else: 0))
|
|
tlits.add trLits[k]
|
|
terr.add trErr[k]
|
|
th.add hi
|
|
tturn.add trTurn[k]
|
|
tlab.add cls
|
|
|
|
proc sideMedian(errs: seq[float]): array[2, float] =
|
|
var l, r: seq[float]
|
|
for e in errs:
|
|
if e > 1e-9: l.add e
|
|
elif e < -1e-9: r.add e
|
|
result[0] = medOf(l)
|
|
result[1] = medOf(r)
|
|
|
|
# ── train TM (fresh per round) ──
|
|
var tm = newMachine(N_BITS, N_CLASSES, cfgClauses, cfgStates, cfgS,
|
|
seed = 12345'u64 + uint64(r.roundNo))
|
|
trainMachine(tm, tlits, tlab, cfgEpochs, seed = 999'u64 + uint64(r.roundNo))
|
|
|
|
# ── calibration on the out-of-sample slice ─────────────────────────────
|
|
var calHalf = newSeq[float](calErr.len)
|
|
for k in 0..<calErr.len:
|
|
calHalf[k] = radToDeg(arctan2(BOT_RADIUS, max(1e-6, calRange[k])))
|
|
let grid = shiftGrid()
|
|
|
|
proc sideOf(e: float): int =
|
|
if e > 1e-9: 0 elif e < -1e-9: 1 else: -1
|
|
|
|
# TM predictions on the calibration slice (computed once, reused below).
|
|
var calPredTM = newSeq[int](calErr.len)
|
|
var calPredShuf = newSeq[int](calErr.len)
|
|
block:
|
|
var cache = newSeq[uint8](tm.nClauses)
|
|
for idx in 0..<calErr.len:
|
|
calPredTM[idx] = predictWith(tm, calLits[idx], cache)
|
|
|
|
# Reference: the OLD gate-2 appliance (conditional MEDIAN per quadrant).
|
|
var repOff: array[NH, array[N_CLASSES, float]]
|
|
block:
|
|
var clsE: array[NH, array[N_CLASSES, seq[float]]]
|
|
for idx in 0..<calErr.len:
|
|
clsE[calH[idx]][calPredTM[idx]].add calErr[idx]
|
|
for hi in 0..<NH:
|
|
let sm2 = sideMedian((block:
|
|
var e: seq[float]
|
|
for idx in 0..<calErr.len:
|
|
if calH[idx] == hi: e.add calErr[idx]
|
|
e))
|
|
for c in 0..<N_CLASSES:
|
|
repOff[hi][c] = if clsE[hi][c].len > 0: medOf(clsE[hi][c])
|
|
else: (if c < 2: sm2[0] else: sm2[1])
|
|
|
|
# HIT-OPTIMAL shift per predicted quadrant (the new appliance).
|
|
block:
|
|
var clsE: array[NH, array[N_CLASSES, seq[float]]]
|
|
var clsH: array[NH, array[N_CLASSES, seq[float]]]
|
|
for idx in 0..<calErr.len:
|
|
let pc = calPredTM[idx]
|
|
clsE[calH[idx]][pc].add calErr[idx]
|
|
clsH[calH[idx]][pc].add calHalf[idx]
|
|
for hi in 0..<NH:
|
|
let sm2 = sideMedian((block:
|
|
var e: seq[float]
|
|
for idx in 0..<calErr.len:
|
|
if calH[idx] == hi: e.add calErr[idx]
|
|
e))
|
|
for c in 0..<N_CLASSES:
|
|
result.fitTM[hi][c] =
|
|
if clsE[hi][c].len > 0: fitHitOptimal(clsE[hi][c], clsH[hi][c], grid)
|
|
else: (if c < 2: sm2[0] else: sm2[1])
|
|
|
|
# HIT-OPTIMAL shift per predicted SIDE (quadrant collapsed) — apples-to-apples
|
|
# with PERF-SIGN: separates "the quadrant split is noisy" from "60% is too low".
|
|
block:
|
|
var sideE: array[NH, array[2, seq[float]]]
|
|
var sideH: array[NH, array[2, seq[float]]]
|
|
for idx in 0..<calErr.len:
|
|
let sd = if calPredTM[idx] <= 1: 0 else: 1
|
|
sideE[calH[idx]][sd].add calErr[idx]
|
|
sideH[calH[idx]][sd].add calHalf[idx]
|
|
for hi in 0..<NH:
|
|
for sd in 0..1:
|
|
result.fitTMSide[hi][sd] =
|
|
if sideE[hi][sd].len > 0: fitHitOptimal(sideE[hi][sd], sideH[hi][sd], grid)
|
|
else: 0.0
|
|
|
|
# ── shuffled-label control ──
|
|
# Permute the fit labels within each horizon, retrain, and apply the SAME
|
|
# hit-optimal appliance to the shuffled model.
|
|
var slab = tlab
|
|
block:
|
|
var rng = seedRng(4242'u64 + uint64(r.roundNo))
|
|
for hi in 0..<NH:
|
|
var pos: seq[int]
|
|
for idx in 0..<tlits.len:
|
|
if th[idx] == hi: pos.add idx
|
|
var perm = pos
|
|
for i in countdown(perm.len - 1, 1):
|
|
let j = int(rng.nextU64() mod uint64(i + 1))
|
|
swap(perm[i], perm[j])
|
|
for i in 0..<pos.len: slab[pos[i]] = tlab[perm[i]]
|
|
|
|
var sm = newMachine(N_BITS, N_CLASSES, cfgClauses, cfgStates, cfgS,
|
|
seed = 777'u64 + uint64(r.roundNo))
|
|
trainMachine(sm, tlits, slab, cfgEpochs, seed = 555'u64 + uint64(r.roundNo))
|
|
|
|
block:
|
|
var cache = newSeq[uint8](sm.nClauses)
|
|
for idx in 0..<calErr.len:
|
|
calPredShuf[idx] = predictWith(sm, calLits[idx], cache)
|
|
|
|
# Shuffled control: SAME hit-optimal appliance on the shuffled model.
|
|
block:
|
|
var clsE: array[NH, array[N_CLASSES, seq[float]]]
|
|
var clsH: array[NH, array[N_CLASSES, seq[float]]]
|
|
for idx in 0..<calErr.len:
|
|
let pc = calPredShuf[idx]
|
|
clsE[calH[idx]][pc].add calErr[idx]
|
|
clsH[calH[idx]][pc].add calHalf[idx]
|
|
for hi in 0..<NH:
|
|
let sm2 = sideMedian((block:
|
|
var e: seq[float]
|
|
for idx in 0..<calErr.len:
|
|
if calH[idx] == hi: e.add calErr[idx]
|
|
e))
|
|
for c in 0..<N_CLASSES:
|
|
result.fitShuf[hi][c] =
|
|
if clsE[hi][c].len > 0: fitHitOptimal(clsE[hi][c], clsH[hi][c], grid)
|
|
else: (if c < 2: sm2[0] else: sm2[1])
|
|
|
|
# ── PERFECT-SIGN + hit-optimal shift (DECISIVE, cheating arm) ──
|
|
# The bucket is the TRUE sign of the residual (not available in reality);
|
|
# the shift is the hit-optimal shift on the calibration slice per true side.
|
|
block:
|
|
var sideE: array[NH, array[2, seq[float]]]
|
|
var sideH: array[NH, array[2, seq[float]]]
|
|
for idx in 0..<calErr.len:
|
|
let sd = sideOf(calErr[idx])
|
|
if sd < 0: continue
|
|
sideE[calH[idx]][sd].add calErr[idx]
|
|
sideH[calH[idx]][sd].add calHalf[idx]
|
|
for hi in 0..<NH:
|
|
for sd in 0..1:
|
|
result.fitPS[hi][sd] =
|
|
if sideE[hi][sd].len > 0: fitHitOptimal(sideE[hi][sd], sideH[hi][sd], grid)
|
|
else: 0.0
|
|
|
|
# side-accuracy sweep: what q does the form need? For each target accuracy,
|
|
# a synthetic predictor correct with probability q (independent draws on the
|
|
# calibration and eval slices); fit per predicted side, apply out-of-sample.
|
|
block:
|
|
for qi, q in ACCS:
|
|
var rng = seedRng(31337'u64 + uint64(r.roundNo) * 131'u64 + uint64(qi))
|
|
var calSide = newSeq[int](calErr.len)
|
|
for idx in 0..<calErr.len:
|
|
let ts = sideOf(calErr[idx])
|
|
if ts < 0: calSide[idx] = if rng.rand01() < 0.5: 0 else: 1
|
|
else: calSide[idx] = if rng.rand01() < q: ts else: 1 - ts
|
|
var sideE: array[NH, array[2, seq[float]]]
|
|
var sideH: array[NH, array[2, seq[float]]]
|
|
for idx in 0..<calErr.len:
|
|
sideE[calH[idx]][calSide[idx]].add calErr[idx]
|
|
sideH[calH[idx]][calSide[idx]].add calHalf[idx]
|
|
var off: array[NH, array[2, float]]
|
|
for hi in 0..<NH:
|
|
for sd in 0..1:
|
|
off[hi][sd] =
|
|
if sideE[hi][sd].len > 0: fitHitOptimal(sideE[hi][sd], sideH[hi][sd], grid)
|
|
else: 0.0
|
|
result.accFit[qi][hi][sd] = off[hi][sd]
|
|
for k in 0..<evErr.len:
|
|
let hi = evH[k]
|
|
let ts = sideOf(evErr[k])
|
|
let pside =
|
|
if ts < 0: (if rng.rand01() < 0.5: 0 else: 1)
|
|
else: (if rng.rand01() < q: ts else: 1 - ts)
|
|
let half = radToDeg(arctan2(BOT_RADIUS, max(1e-6, evRange[k])))
|
|
result.accN[qi][hi] += 1
|
|
if abs(evErr[k] - off[hi][pside]) < half: inc result.accHits[qi][hi]
|
|
|
|
# hits-vs-constant-shift diagnostic curve, pooled calibration samples/horizon
|
|
for idx in 0..<calErr.len:
|
|
let hi = calH[idx]
|
|
inc result.curveN[hi]
|
|
for k in 0..<NCURVE:
|
|
if abs(calErr[idx] - curveShift(k)) < calHalf[idx]:
|
|
inc result.curveHits[hi][k]
|
|
|
|
# ── evaluate ──
|
|
var sc = newSeq[uint8](sm.nClauses)
|
|
var tc = newSeq[uint8](tm.nClauses)
|
|
for k in 0..<evErr.len:
|
|
let hi = evH[k]
|
|
let i = evIdx[k]
|
|
let err = evErr[k]
|
|
let rng = evRange[k]
|
|
result.n[hi] += 1
|
|
let half = radToDeg(arctan2(BOT_RADIUS, max(1e-6, rng)))
|
|
|
|
let lits = toLits(base[i], HORIZONS[hi])
|
|
let predTM = predictWith(tm, lits, tc)
|
|
let predShuf = predictWith(sm, lits, sc)
|
|
let ps = sideOf(err) # TRUE side (cheating, only for aPS/aPS0)
|
|
|
|
let offs: array[Arm, float] = [
|
|
aNaive: 0.0,
|
|
aTMmed: repOff[hi][predTM],
|
|
aTMhit: result.fitTM[hi][predTM],
|
|
aTMside: result.fitTMSide[hi][(if predTM <= 1: 0 else: 1)],
|
|
aTMs0: 0.0,
|
|
aTMs25: 0.25 * result.fitTM[hi][predTM],
|
|
aTMs50: 0.50 * result.fitTM[hi][predTM],
|
|
aPS: (if ps >= 0: result.fitPS[hi][ps] else: 0.0),
|
|
aPS0: 0.0,
|
|
aShuf: result.fitShuf[hi][predShuf]]
|
|
|
|
# diagnostics on the SAME eval tick
|
|
let trueCls = ((if err > 0.0: 0 else: 2) +
|
|
(if abs(err) > medAbs[hi]: 1 else: 0))
|
|
inc result.trueCounts[hi][trueCls]
|
|
inc result.predCounts[hi][predTM]
|
|
inc result.tmQTot[hi]
|
|
if predTM == trueCls: inc result.tmQCor[hi]
|
|
inc result.shufQTot[hi]
|
|
if predShuf == trueCls: inc result.shufQCor[hi]
|
|
inc result.tmHitTot[hi]
|
|
if (predTM <= 1) == (trueCls <= 1): inc result.tmHitCor[hi]
|
|
|
|
for a in Arm:
|
|
let res = err - offs[a]
|
|
result.errs[a][hi].add abs(res)
|
|
if abs(res) < half: inc result.hits[a][hi]
|
|
|
|
# ── per-round table ──
|
|
if emit:
|
|
echo &" round {r.roundNo:>2} (L={L:>5}) " &
|
|
"nEval/h = " & $[result.n[0], result.n[1], result.n[2], result.n[3]]
|
|
echo " h arm med p90 hit%"
|
|
for hi in 0..<NH:
|
|
for a in ShowArms:
|
|
let m = medOf(result.errs[a][hi])
|
|
let p = qOf(result.errs[a][hi], 0.9)
|
|
let hit = if result.n[hi] > 0: 100.0 * result.hits[a][hi].float / result.n[hi].float else: NaN
|
|
echo &" {HORIZONS[hi]:>3} {ArmName[a]:<9} {m:>6.2f} {p:>6.2f} {hit:>6.1f}"
|
|
|
|
# ── table printing ───────────────────────────────────────────────────────────
|
|
|
|
proc printPooled(h: Hist, title: string) =
|
|
echo "\n" & title
|
|
echo " h arm N med|err| p90|err| hit% miss%"
|
|
for hi in 0..<NH:
|
|
let n = h.n[hi]
|
|
for a in Arm:
|
|
let m = medOf(h.errs[a][hi])
|
|
let p = qOf(h.errs[a][hi], 0.9)
|
|
let hit = if n > 0: 100.0 * h.hits[a][hi].float / n.float else: NaN
|
|
echo &"{HORIZONS[hi]:>3} {ArmName[a]:<9} {n:>7} {m:>9.2f} {p:>9.2f} {hit:>7.1f} {100.0-hit:>7.1f}"
|
|
|
|
proc printDeltas(h: Hist, title: string) =
|
|
echo "\n" & title
|
|
echo " h naive% dTMmed dTMhit dTMside dTMs0 dTMs25 dTMs50 dPS dPS0 dShuf (hit pp vs naive)"
|
|
for hi in 0..<NH:
|
|
let n = h.n[hi].float
|
|
if n <= 0: continue
|
|
let base = 100.0 * h.hits[aNaive][hi].float / n
|
|
var row = &"{HORIZONS[hi]:>3} {base:>6.1f} "
|
|
for a in Arm:
|
|
if a == aNaive: continue
|
|
let v = 100.0 * h.hits[a][hi].float / n - base
|
|
row.add &"{v:>+6.1f} "
|
|
echo row
|
|
|
|
proc printDiagnostics(h: Hist, title: string) =
|
|
## Is the learner actually learning? (quadrant / side accuracy vs chance)
|
|
echo "\n" & title
|
|
echo " h N TMquad% shufquad% TMside% majQuad% pred[Ls Lb Rs Rb] / true[Ls Lb Rs Rb]"
|
|
for hi in 0..<NH:
|
|
let n = h.n[hi]
|
|
if n == 0: continue
|
|
let tmq = 100.0 * float(h.tmQCor[hi]) / float(n)
|
|
let shufq = 100.0 * float(h.shufQCor[hi]) / float(max(1, h.shufQTot[hi]))
|
|
let tms = 100.0 * float(h.tmHitCor[hi]) / float(n)
|
|
var mx = 0
|
|
for c in 0..<N_CLASSES: mx = max(mx, h.trueCounts[hi][c])
|
|
let maj = 100.0 * float(mx) / float(n)
|
|
var pcs = ""
|
|
var tcs = ""
|
|
for c in 0..<N_CLASSES:
|
|
pcs.add $h.predCounts[hi][c] & " "
|
|
tcs.add $h.trueCounts[hi][c] & " "
|
|
echo &"{HORIZONS[hi]:>3} {n:>6} {tmq:>8.1f} {shufq:>10.1f} {tms:>8.1f} {maj:>9.1f} [{pcs}] / [{tcs}]"
|
|
|
|
proc printFittedSummary(rounds: seq[Hist], title: string) =
|
|
## Median across rounds of the fitted hit-optimal shift (deg).
|
|
echo "\n" & title
|
|
echo " (median across rounds of the fitted HIT-OPTIMAL shift, deg)"
|
|
echo " h TM[Ls Lb Rs Rb] PS[left right] shuf[Ls Lb Rs Rb]"
|
|
for hi in 0..<NH:
|
|
var tmC: array[N_CLASSES, seq[float]]
|
|
var psC: array[2, seq[float]]
|
|
var shC: array[N_CLASSES, seq[float]]
|
|
for rh in rounds:
|
|
for c in 0..<N_CLASSES: tmC[c].add rh.fitTM[hi][c]
|
|
for sd in 0..<2: psC[sd].add rh.fitPS[hi][sd]
|
|
for c in 0..<N_CLASSES: shC[c].add rh.fitShuf[hi][c]
|
|
var s1, s2, s3: string
|
|
for c in 0..<N_CLASSES: s1.add &"{medOf(tmC[c]):>6.2f} "
|
|
for sd in 0..<2: s2.add &"{medOf(psC[sd]):>7.2f} "
|
|
for c in 0..<N_CLASSES: s3.add &"{medOf(shC[c]):>6.2f} "
|
|
echo &"{HORIZONS[hi]:>3} [{s1}] [{s2}] [{s3}]"
|
|
|
|
proc printCurve(h: Hist, title: string) =
|
|
## Hit fraction as a function of a CONSTANT shift applied to every
|
|
## calibration sample (pooled across buckets), per horizon.
|
|
echo "\n" & title
|
|
echo " shift h=15 h=20 h=25 h=30 (hit% on calibration, pooled)"
|
|
for k in 0..<NCURVE:
|
|
var row = &"{curveShift(k):>+6.1f} "
|
|
for hi in 0..<NH:
|
|
let n = h.curveN[hi]
|
|
let v = if n > 0: 100.0 * h.curveHits[hi][k].float / n.float else: NaN
|
|
row.add &"{v:>7.1f} "
|
|
echo row
|
|
|
|
proc printAccuracySweep(rounds: seq[Hist], title: string) =
|
|
## Gain (pp vs naive) for a synthetic side predictor of accuracy q.
|
|
echo "\n" & title
|
|
echo " q fitShift(h=20)[L R] hit% h15 h20 h25 h30 d(hit) pp: h15 h20 h25 h30 mean"
|
|
var naiveHits, naiveN: array[NH, int]
|
|
for rh in rounds:
|
|
for hi in 0..<NH:
|
|
naiveHits[hi] += rh.hits[aNaive][hi]
|
|
naiveN[hi] += rh.n[hi]
|
|
for qi, q in ACCS:
|
|
var sh: array[2, seq[float]]
|
|
var hits, nn: array[NH, int]
|
|
for rh in rounds:
|
|
for sd in 0..1: sh[sd].add rh.accFit[qi][1][sd]
|
|
for hi in 0..<NH:
|
|
hits[hi] += rh.accHits[qi][hi]
|
|
nn[hi] += rh.accN[qi][hi]
|
|
var dRow, hRow: string
|
|
var dsum = 0.0
|
|
for hi in 0..<NH:
|
|
let hp = if nn[hi] > 0: 100.0 * hits[hi].float / nn[hi].float else: NaN
|
|
let np = if naiveN[hi] > 0: 100.0 * naiveHits[hi].float / naiveN[hi].float else: NaN
|
|
hRow.add &"{hp:>6.1f}"
|
|
dRow.add &"{hp - np:>+6.1f}"
|
|
dsum += hp - np
|
|
echo &" {q:>4.2f} [{medOf(sh[0]):>6.2f} {medOf(sh[1]):>6.2f}] {hRow} {dRow} {dsum/float(NH):>+6.1f}"
|
|
|
|
# ── main ─────────────────────────────────────────────────────────────────────
|
|
|
|
proc main() =
|
|
var names = @PRIMARY
|
|
for i in 1..paramCount():
|
|
let a = paramStr(i)
|
|
if a.startsWith("--fixtures="): names = a[11..^1].split(',')
|
|
elif a.startsWith("--epochs="): cfgEpochs = parseInt(a[9..^1])
|
|
elif a.startsWith("--trainfrac="): cfgTrainFrac = parseFloat(a[12..^1])
|
|
elif a.startsWith("--clauses="): cfgClauses = parseInt(a[10..^1])
|
|
|
|
echo "=" .repeat(90)
|
|
echo "GATE 2b - is the shift-per-bucket appliance STRUCTURALLY DEAD or MISCALIBRATED?"
|
|
echo "=" .repeat(90)
|
|
echo &"fixtures : {names.join(\", \")}"
|
|
echo &"bits : {N_BITS} ({N_BASE} draft + 4 horizon one-hot)"
|
|
echo &"TM : {cfgClauses} clauses, {cfgStates} states, s={cfgS}, " &
|
|
&"{cfgEpochs} epochs, fresh per round"
|
|
echo &"split : within-round, train 50%, calibrate {cfgTrainFrac*100:.0f}%, eval rest"
|
|
echo "appliance : shift on {-20..+20 deg, 0.5 step} maximising HIT COUNT"
|
|
echo &" on the calibration slice; applied out-of-sample"
|
|
echo &"estimated hit : |residual| < atan(18px / range) -- ABSOLUTE IS OPTIMISTIC"
|
|
echo &"DECISIVE arm : PERF-SIGN uses the TRUE sign (cheating) + hit-optimal shift"
|
|
echo &"bullet block : INFERRED from self-energy drops (no gun heading recorded)"
|
|
echo "=" .repeat(90)
|
|
|
|
var pooled = Hist()
|
|
var pooledByFix = initTable[string, Hist]()
|
|
var roundHists = initTable[string, seq[Hist]]()
|
|
var allRounds: seq[Hist]
|
|
|
|
for name in names:
|
|
let path = if name.endsWith(".jsonl"): fixturesDir / name
|
|
else: fixturesDir / (name & ".jsonl")
|
|
if not fileExists(path):
|
|
echo &"# SKIP missing fixture {path}"
|
|
continue
|
|
let ticks = loadTicks(path)
|
|
let rounds = loadRounds(path, ticks)
|
|
echo &"\n## FIXTURE {name}: {rounds.len} rounds, {ticks.len} ticks"
|
|
var fhist = Hist()
|
|
var rlist: seq[Hist]
|
|
for r in rounds:
|
|
let bi = buildBulletSeries(r)
|
|
var cache = newSeq[uint8](cfgClauses)
|
|
let rh = runRound(r, bi, cache, emit = true)
|
|
mergeHist(fhist, rh)
|
|
mergeHist(pooled, rh)
|
|
rlist.add rh
|
|
allRounds.add rh
|
|
echo ""
|
|
pooledByFix[name] = fhist
|
|
roundHists[name] = rlist
|
|
printPooled(fhist, &"## PER-FIXTURE pooled ({name})")
|
|
printDiagnostics(fhist, &"## PER-FIXTURE learnability ({name})")
|
|
|
|
echo "\n" & "=".repeat(90)
|
|
printPooled(pooled, "## POOLED PRIMARY (tr_drussgt_vs_modularbot*): arm table")
|
|
printDeltas(pooled, "## DELTAS (percentage points of estimated hit fraction)")
|
|
printDiagnostics(pooled, "## POOLED learnability diagnostics (is the learner learning?)")
|
|
printFittedSummary(allRounds, "## FITTED HIT-OPTIMAL SHIFT MAGNITUDES")
|
|
printCurve(pooled, "## HIT FRACTION vs CONSTANT SHIFT MAGNITUDE (calibration)")
|
|
printAccuracySweep(allRounds, "## SIDE-ACCURACY SWEEP (synthetic q-accurate predictor, out-of-sample)")
|
|
|
|
echo "\n" & "=".repeat(90)
|
|
echo "## PER-ROUND hit-delta integrity (pp vs naive)"
|
|
echo "fixture round h TMmed TMhit TMside TMs25 TMs50 PS shuf"
|
|
for name in names:
|
|
if not roundHists.hasKey(name): continue
|
|
for rn, rh in roundHists[name]:
|
|
for hi in 0..<NH:
|
|
let n = rh.n[hi].float
|
|
if n <= 0: continue
|
|
let hitNaive = 100.0 * rh.hits[aNaive][hi].float / n
|
|
let dTMmed = 100.0 * rh.hits[aTMmed][hi].float / n - hitNaive
|
|
let dTMhit = 100.0 * rh.hits[aTMhit][hi].float / n - hitNaive
|
|
let dTMside = 100.0 * rh.hits[aTMside][hi].float / n - hitNaive
|
|
let dTMs25 = 100.0 * rh.hits[aTMs25][hi].float / n - hitNaive
|
|
let dTMs50 = 100.0 * rh.hits[aTMs50][hi].float / n - hitNaive
|
|
let dPS = 100.0 * rh.hits[aPS][hi].float / n - hitNaive
|
|
let dShuf = 100.0 * rh.hits[aShuf][hi].float / n - hitNaive
|
|
echo &"{name:<34} {rn+1:>5} {HORIZONS[hi]:>3} {dTMmed:>+6.1f} " &
|
|
&"{dTMhit:>+6.1f} {dTMside:>+6.1f} {dTMs25:>+6.1f} {dTMs50:>+6.1f} " &
|
|
&"{dPS:>+6.1f} {dShuf:>+6.1f}"
|
|
|
|
# ── unhedged verdict inputs ──
|
|
echo "\n" & "=".repeat(90)
|
|
echo "## VERDICT INPUTS (pooled hit pp vs naive)"
|
|
for hi in 0..<NH:
|
|
let n = pooled.n[hi].float
|
|
if n <= 0: continue
|
|
let base = 100.0 * pooled.hits[aNaive][hi].float / n
|
|
let ps = 100.0 * pooled.hits[aPS][hi].float / n - base
|
|
let tm = 100.0 * pooled.hits[aTMhit][hi].float / n - base
|
|
let tms = 100.0 * pooled.hits[aTMside][hi].float / n - base
|
|
let sh = 100.0 * pooled.hits[aShuf][hi].float / n - base
|
|
echo &"h={HORIZONS[hi]:>3} PERF-SIGN {ps:>+6.1f} pp TM-side {tms:>+6.1f} pp TM-quad {tm:>+6.1f} pp shuffled {sh:>+6.1f} pp"
|
|
|
|
when isMainModule:
|
|
main()
|