Radial offset: STRUCTURALLY incapable of helping, and Pattern does not overshoot
Follow-up to 9cd6e9b, which found the LINEAR base systematically overshoots (mean
radial error -71..-100px, enemy nearer in 63-81% of shots). The question was
whether the gun that actually ships, `Pattern`, overshoots too - because
correcting a systematic bias would be a cheap win.
1. PATTERN DOES NOT OVERSHOOT. Measured over the DrussGT fixtures (n=250,989):
Pattern mean -12.0 px, median -3.2 px, nearer 52.8% / farther 45.0%
Linear mean -87.3 px, median -61.0 px, nearer 83.4% / farther 14.4% (same states)
So the overshoot was a property of the CONSTANT-VELOCITY BASE, not of our
predictions in general. Pattern's pattern-matching does not have it, so there
was nothing to correct. (All-10-fixture pooled: mean -14.0, median -4.2.)
2. THE AVENUE IS STRUCTURALLY DEAD, not merely unprofitable. The live aim is
`aimAngle(self, pred)` and a RADIAL-only offset keeps the BEARING unchanged
(proven exactly by a guard test: bearing is invariant). So a radial offset
cannot change the fired bullet's direction at all. `bmPath` never scores the
aim distance either - and measured, every offset arm is BYTE-IDENTICAL to plain
Pattern on bmPath (33.9%/25.6%). The only real-effect channel is the `shouldFire`
gate via `distPx`, which is indistinguishable from noise.
3. LIVE A/B CONFIRMS: one frozen binary (built from HEAD + only this change),
env-only arms, 7 runs x 7 rounds, 8 concurrent, real DrussGT, server-side hit
rate, exact two-sided permutation test.
control (plain Pattern) 10.61% / 284 dmg-per-run
s0.98 10.89% / 302 (+0.28pp, p=0.62)
s0.95 10.02% (p=0.35)
o-20 10.19% (p=0.46)
No significant winner.
VERDICT: STOP. This line cannot help the shipped configuration, and the reason is
structural rather than statistical - a radial correction is bearing-invariant, so
it is invisible to the actual shot. The bmPoint "win" the radial TM showed was a
metric artefact of that same irrelevance.
Incidental: the control arm (10.61% / 284) independently replicates the shipped
Pattern-only default's A/B numbers (10.36% / 264, 10.78% / 287, 9.99%).
Kept anyway: `TR_PATTERN_RAD_SCALE` / `TR_PATTERN_RAD_OFFSET` default to
(1.0, 0.0) and the default path is byte-identical (proven over 2400 predictions,
plus bearing invariance and unparsable-value fallback - 6 checks). Adds
measure_pattern_radial.nim, sweep_pattern_radial.nim, test_pattern_radial_offset.nim
and pattern_radial_results.md.
Guards: test_gun_harness 39, test_vbullet_metric 11, test_power_selection 3,
test_adaptive_radar 41, test_tfil_ring_weights 24, test_power_policy 26,
test_ram_decision 28, test_rack_membership 48, test_tm_pattern_registration 20,
acceptance_offline_vs_online 12/12 PASS.
This commit is contained in:
@@ -0,0 +1,211 @@
|
||||
## MEASUREMENT: does the SHIPPED `Pattern` gun (guns/pattern_matcher.nim, rack
|
||||
## gun id 5) overshoot radially the way the LINEAR base was shown to?
|
||||
##
|
||||
## Background (commit 9cd6e9b): the base `forecastLinear` systematically
|
||||
## OVERSHOOTS — the enemy is NEARER than the constant-velocity prediction in
|
||||
## 63-81% of fired bullets and farther in only 4-14%, consistently across all
|
||||
## six DrussGT captures. A constant short offset (aim distance x0.95 or a fixed
|
||||
## -20 px) matched or beat the learned radial Tsetlin head on bmPoint.
|
||||
##
|
||||
## The gun that actually SHIPS is `Pattern`, not `Linear`. This tool measures the
|
||||
## SAME signed radial error for Pattern: for every fired virtual bullet,
|
||||
##
|
||||
## radialErr = |actual enemy pos at the base arrival tick - fire pos|
|
||||
## - |Pattern's predicted aim point - fire pos|
|
||||
##
|
||||
## (negative = the enemy was NEARER than Pattern predicted = Pattern OVERSHOT).
|
||||
## The arrival tick mirrors the virtual-bullet resolver exactly:
|
||||
## arrOff = max(0, ceil(aimDist / bulletSpeed) - 1), arrivalTick = fireTick + arrOff
|
||||
## because the tracker advances travelDist by bulletSpeed on the spawn tick and
|
||||
## resolves when travelDist >= fireDist.
|
||||
##
|
||||
## It also reports the raw `forecastLinear` error on the identical states, so a
|
||||
## reader can see whether Pattern is better or worse than the base on the same
|
||||
## measurement, and pools per power bin as well as overall.
|
||||
##
|
||||
## READ-ONLY: loads tools/fixtures/*drussgt*.jsonl; writes nothing.
|
||||
##
|
||||
## Run:
|
||||
## nim c -r -d:release --path:common_libs \
|
||||
## common_libs/tests/measure_pattern_radial.nim
|
||||
## Flags: --set=core|all (default core = the six captures of the 9cd6e9b table)
|
||||
|
||||
import std/[os, strformat, strutils, json, tables, math, algorithm]
|
||||
import gun_harness/offline_range
|
||||
import gun_harness/gun_interface
|
||||
import gun_harness/virtual_bullets as vb
|
||||
import guns/pattern_matcher
|
||||
import guns/lead_forecast
|
||||
|
||||
const
|
||||
repoRoot = currentSourcePath().parentDir.parentDir.parentDir
|
||||
fixturesDir = repoRoot / "tools" / "fixtures"
|
||||
|
||||
type
|
||||
RoundSpan = tuple[start, count: int]
|
||||
|
||||
Agg = object
|
||||
n: int
|
||||
sum, asum: float
|
||||
errs: seq[float]
|
||||
near0, far0: int ## err < 0 / err > 0
|
||||
nearBot, farBot: int ## err < -BotRadius / err > +BotRadius
|
||||
|
||||
proc loadRounds(path: string): seq[RoundSpan] =
|
||||
let dir = path.parentDir
|
||||
let base = path.extractFilename
|
||||
var side = dir / "drussgt_meta" / (base & ".rounds.json")
|
||||
if not fileExists(side): side = dir / (base & ".rounds.json")
|
||||
if not fileExists(side): return @[]
|
||||
let node = parseJson(readFile(side))
|
||||
if not node.hasKey("rounds"): return @[]
|
||||
for r in node["rounds"]:
|
||||
result.add (r["startTick"].getInt(), r["count"].getInt())
|
||||
|
||||
proc addErr(a: var Agg, e: float) =
|
||||
inc a.n
|
||||
a.sum += e
|
||||
a.asum += abs(e)
|
||||
a.errs.add e
|
||||
if e < 0.0: inc a.near0
|
||||
elif e > 0.0: inc a.far0
|
||||
if e < -BotRadius: inc a.nearBot
|
||||
elif e > BotRadius: inc a.farBot
|
||||
|
||||
proc median(a: Agg): float =
|
||||
if a.errs.len == 0: return 0.0
|
||||
var s = a.errs
|
||||
s.sort()
|
||||
let m = s.len div 2
|
||||
if s.len mod 2 == 1: s[m] else: 0.5 * (s[m - 1] + s[m])
|
||||
|
||||
proc percentile(a: Agg, p: float): float =
|
||||
if a.errs.len == 0: return 0.0
|
||||
var s = a.errs
|
||||
s.sort()
|
||||
let idx = clamp(int(round(p / 100.0 * float(s.len - 1))), 0, s.len - 1)
|
||||
s[idx]
|
||||
|
||||
proc reportScope(scope: string, agg: var Agg) =
|
||||
if agg.n == 0:
|
||||
echo &"{scope},0,n/a,n/a,n/a,n/a,n/a,n/a"
|
||||
return
|
||||
let p10 = agg.percentile(10)
|
||||
let p50 = agg.median
|
||||
let p90 = agg.percentile(90)
|
||||
echo &"{scope},{agg.n},{agg.sum/float(agg.n):.2f},{p50:.2f},{agg.asum/float(agg.n):.2f}," &
|
||||
&"{100.0*float(agg.near0)/float(agg.n):.1f},{100.0*float(agg.far0)/float(agg.n):.1f}," &
|
||||
&"{100.0*float(agg.nearBot)/float(agg.n):.1f},{100.0*float(agg.farBot)/float(agg.n):.1f}," &
|
||||
&"{p10:.1f},{p90:.1f}"
|
||||
|
||||
proc coreSet(): seq[string] =
|
||||
## The exact six captures of the committed 9cd6e9b base table, so the Pattern
|
||||
## numbers are directly comparable.
|
||||
for n in ["drussgt_vs_crazy", "drussgt_vs_spinbot", "drussgt_vs_drussgt",
|
||||
"tr_drussgt_vs_crazy", "tr_drussgt_vs_spinbot",
|
||||
"tr_drussgt_vs_modularbot"]:
|
||||
result.add(fixturesDir / (n & ".jsonl"))
|
||||
|
||||
proc allSet(): seq[string] =
|
||||
for kind, p in walkDir(fixturesDir):
|
||||
if kind == pcFile and p.extractFilename.contains("drussgt") and
|
||||
p.extractFilename.endsWith(".jsonl"):
|
||||
result.add p
|
||||
result.sort()
|
||||
|
||||
proc main() =
|
||||
var which = "core"
|
||||
for i in 1..paramCount():
|
||||
let a = paramStr(i)
|
||||
if a.startsWith("--set="): which = a[6..^1]
|
||||
let names = if which == "all": allSet() else: coreSet()
|
||||
|
||||
echo "# Patterns radial error — negative = enemy NEARER than predicted (OVERSHOOT)"
|
||||
echo "# arrivalTick = fireTick + max(0, ceil(aimDist/speed) - 1); aimDist from the gun's own (px,py)"
|
||||
echo "# scope,n,meanPx,medianPx,meanAbsPx,pctNearer(<0),pctFarther(>0),pctNearer(<-18px),pctFarther(>+18px),p10,p90"
|
||||
|
||||
var patAll: Agg
|
||||
var linAll: Agg
|
||||
var patCore: Agg
|
||||
var linCore: Agg
|
||||
let coreNames = coreSet()
|
||||
|
||||
for path in names:
|
||||
if not fileExists(path): continue
|
||||
let fx = loadFixture(path)
|
||||
let fxName = path.extractFilename.replace(".jsonl", "")
|
||||
var pat: Agg
|
||||
var lin: Agg
|
||||
# inline replay so we keep the two errors paired
|
||||
var pose = initTable[int, tuple[x, y: float]]()
|
||||
for s in fx.states: pose[s.tick] = (s.enemyX, s.enemyY)
|
||||
var spans = loadRounds(path)
|
||||
if spans.len == 0:
|
||||
spans = @[(start: fx.states[0].tick, count: fx.states.len)]
|
||||
for sp in spans:
|
||||
var g = PatternMatcherGun()
|
||||
for i in 0..<fx.states.len:
|
||||
let state = fx.states[i]
|
||||
if state.tick < sp.start or state.tick >= sp.start + sp.count: continue
|
||||
for b in 0..<len(vb.PowerBins):
|
||||
let speed = bulletSpeed(vb.PowerBins[b])
|
||||
let pred = g.predict(state, speed)
|
||||
let aimDist = hypot(pred.x - state.selfX, pred.y - state.selfY)
|
||||
let at = state.tick + max(0, int(ceil(aimDist / speed)) - 1)
|
||||
if at in pose and aimDist > 1e-9:
|
||||
let actualR = hypot(pose[at].x - state.selfX, pose[at].y - state.selfY)
|
||||
addErr(pat, actualR - aimDist)
|
||||
let f = forecastLinear(state, speed)
|
||||
let lArr = state.tick + max(0, int(ceil(f.dist / speed)) - 1)
|
||||
if lArr in pose:
|
||||
let actualR = hypot(pose[lArr].x - state.selfX, pose[lArr].y - state.selfY)
|
||||
addErr(lin, actualR - f.dist)
|
||||
reportScope("PATTERN_" & fxName, pat)
|
||||
reportScope("LINEAR_" & fxName, lin)
|
||||
|
||||
if pat.n > 0:
|
||||
inc patAll.n, pat.n
|
||||
patAll.sum += pat.sum; patAll.asum += pat.asum
|
||||
patAll.near0 += pat.near0; patAll.far0 += pat.far0
|
||||
patAll.nearBot += pat.nearBot; patAll.farBot += pat.farBot
|
||||
for e in pat.errs: patAll.errs.add e
|
||||
if lin.n > 0:
|
||||
inc linAll.n, lin.n
|
||||
linAll.sum += lin.sum; linAll.asum += lin.asum
|
||||
linAll.near0 += lin.near0; linAll.far0 += lin.far0
|
||||
linAll.nearBot += lin.nearBot; linAll.farBot += lin.farBot
|
||||
for e in lin.errs: linAll.errs.add e
|
||||
if path in coreNames:
|
||||
inc patCore.n, pat.n
|
||||
patCore.sum += pat.sum; patCore.asum += pat.asum
|
||||
patCore.near0 += pat.near0; patCore.far0 += pat.far0
|
||||
patCore.nearBot += pat.nearBot; patCore.farBot += pat.farBot
|
||||
for e in pat.errs: patCore.errs.add e
|
||||
inc linCore.n, lin.n
|
||||
linCore.sum += lin.sum; linCore.asum += lin.asum
|
||||
linCore.near0 += lin.near0; linCore.far0 += lin.far0
|
||||
linCore.nearBot += lin.nearBot; linCore.farBot += lin.farBot
|
||||
for e in lin.errs: linCore.errs.add e
|
||||
|
||||
echo ""
|
||||
reportScope("PATTERN_CORE_POOLED", patCore)
|
||||
reportScope("LINEAR_CORE_POOLED", linCore)
|
||||
reportScope("PATTERN_ALL_POOLED", patAll)
|
||||
reportScope("LINEAR_ALL_POOLED", linAll)
|
||||
|
||||
# distribution of the pooled core Pattern error
|
||||
echo "\n# Pattern CORE pooled error distribution (20px bins)"
|
||||
var hist = initOrderedTable[string, int]()
|
||||
let edges = [-1e9, -150.0, -100.0, -60.0, -20.0, 20.0, 60.0, 100.0, 150.0, 1e9]
|
||||
for e in patCore.errs:
|
||||
var label = "?"
|
||||
for k in 0..<edges.len-1:
|
||||
if e >= edges[k] and e < edges[k+1]:
|
||||
label = &"[{edges[k]:.0f},{edges[k+1]:.0f})"
|
||||
break
|
||||
hist[label] = hist.getOrDefault(label) + 1
|
||||
for label, c in hist:
|
||||
echo &" {label:<18} {c:>7} {100.0*float(c)/float(max(1,patCore.n)):5.1f}%"
|
||||
|
||||
when isMainModule:
|
||||
main()
|
||||
Reference in New Issue
Block a user