TM verdict, settled: it loses LIVE and sits at/below its majority class - (c)
The user pushed back on "the TM can't be your best 1v1 gun", correctly, because two
decisive tests had never been run. Both are now run and they agree.
TASK 1 - THE GF HEAD vs ITS MAJORITY-CLASS BASELINE (offline, n=1,751,067):
label histogram [254286, 284578, 678879, 297055, 236269]
majority class = 2 (the CENTRE bucket) = 38.77%
RAW head accuracy = 36.69% -> margin **-2.08 pp, BELOW majority**
GATED head accuracy = 40.37% vs 38.75% majority -> +1.62 pp, BUT it predicts the
majority class on 62.4% of ticks and its minority recall is 13.6% / 12.9% - a
base-rate predictor wearing a classifier's clothes.
Shuffled control sits at its own majority (20.04% vs 20.12%), confirming chance.
**THE OLD "46% vs 20% CHANCE" FIGURE I QUOTED WAS WRONG ON TWO COUNTS:** the
baseline is 38.8%, not 20%, and the 46% predated the deferred-label fix. Against
the correct baseline the head is BELOW it.
TASK 2 - THE FIRST-EVER LIVE A/B OF THE TM GUN (7 runs x 7 rounds per arm, one
frozen binary from git archive HEAD = eb74f9b2, sha256 cb66d66b..., real DrussGT,
every arm forced alone with TR_RACK_<GUN>=both and all 14 others off, liveness
confirmed per run):
arm shots real % dmg/run round wins
onlyPattern 4610 10.74% 285 25/49
onlyTMPATTERN (radial) 3374 3.50% 71 0/49
onlyLinear 3218 3.23% 61 0/49
Pattern vs TM: +7.22 pp / +213.7 dmg, exact p=0.0006
TM vs Linear: +0.30 pp, p=0.659 (dmg p=0.438)
**The TM is statistically INDISTINGUISHABLE from its own Linear base live.** So it
is not "the TM works and we are aiming it wrong".
DIRECT ANSWER: **(c) It loses live AND sits at/below majority - the target carries
no learnable signal beyond the base rate, and that is the reason.** The reason is
not the machine, not the knobs, and not the application alone: the thing it was
asked to predict is dominated by the modal answer.
This closes the TM-as-gun thread. If a TM is wanted in the bot, a firing gate or a
movement decision is a better fit for a boolean-rule classifier than an aim point -
that is untested and is a different project.
A LIVE GF-MODE ARM WAS NOT RUN (stated as unmeasured): the task pinned one frozen
HEAD binary and HEAD registers the TM gun as radial only; Task 1 already makes GF
the unpromising candidate.
HARNESS FIX WORTH KEEPING: `tools/ab/which_gun_arm_env.sh` left the TARGET gun
unset, so with the now-Pattern-only default it silently fell back to the FULL rack
- an arm could appear to test a single gun while actually running the whole rack.
It now emits `TR_RACK_<GUN>=both` for the target and `=off` for all 14 others.
(Earlier which-gun results are unaffected: they ran before the Pattern-only default,
or - as in the melee/1v1 campaign - set the explicit `=both` themselves.)
tm_pattern.nim gains a per-class confusion matrix (warm samples only) to support the
majority baseline; no behaviour change. Adds Round 4 to
tm_pattern_sweep_results.md with both tasks and the interpretation rule.
This commit is contained in:
@@ -1,13 +1,20 @@
|
||||
# emits `TR_RACK_<GUN>=off ...` for every gun NOT in the arm.
|
||||
# usage: arm_env.sh <full|onlyLinear|onlyPattern|onlyGF|onlyKNN|lean8|lean6|pairPC|pairPK|pairPL>
|
||||
# emits `TR_RACK_<GUN>=both` for the kept gun and `TR_RACK_<GUN>=off` for every
|
||||
# other gun, so the rack can never silently fall back to the FULL-rack
|
||||
# degradation when the target gun's `=both` was forgotten.
|
||||
# usage: arm_env.sh <full|onlyLinear|onlyPattern|onlyTMPATTERN|onlyGF|onlyKNN|lean8|lean6|pairPC|pairPK|pairPL>
|
||||
arm="$1"
|
||||
GUNS="HEADON LINEAR TSETLIN CIRCULAR GUESSFACTOR PATTERN WALLBOUNCE ACCEL STOPSHOT DISPLACE AVGLEAD DECAYGF KNN TMSELECT"
|
||||
GUNS="HEADON LINEAR TSETLIN CIRCULAR GUESSFACTOR PATTERN WALLBOUNCE ACCEL STOPSHOT DISPLACE AVGLEAD DECAYGF KNN TMSELECT TMPATTERN"
|
||||
if [ "$arm" = "full" ]; then
|
||||
# Every gun `both` (the process default is onlyPattern, so this must be
|
||||
# explicit).
|
||||
keep="$GUNS"
|
||||
else
|
||||
case "$arm" in
|
||||
full) keep="";;
|
||||
onlyLinear) keep="LINEAR";;
|
||||
onlyPattern) keep="PATTERN";;
|
||||
onlyGF) keep="GUESSFACTOR";;
|
||||
onlyKNN) keep="KNN";;
|
||||
onlyLinear) keep="LINEAR";;
|
||||
onlyPattern) keep="PATTERN";;
|
||||
onlyTMPATTERN) keep="TMPATTERN";;
|
||||
onlyGF) keep="GUESSFACTOR";;
|
||||
onlyKNN) keep="KNN";;
|
||||
# ── small-good-rack follow-up ──────────────────────────────────────────────
|
||||
lean8) keep="HEADON LINEAR CIRCULAR ACCEL PATTERN GUESSFACTOR KNN WALLBOUNCE";;
|
||||
lean6) keep="LINEAR CIRCULAR ACCEL PATTERN GUESSFACTOR KNN";;
|
||||
@@ -16,11 +23,12 @@ case "$arm" in
|
||||
pairPL) keep="PATTERN LINEAR";;
|
||||
*) echo "unknown arm $arm" >&2; exit 1;;
|
||||
esac
|
||||
fi
|
||||
out=""
|
||||
for g in $GUNS; do
|
||||
case " $keep " in
|
||||
*" $g "*) continue;;
|
||||
*" $g "*) out="$out TR_RACK_${g}=both";;
|
||||
*) out="$out TR_RACK_${g}=off";;
|
||||
esac
|
||||
out="$out TR_RACK_${g}=off"
|
||||
done
|
||||
echo "$out"
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
set -u
|
||||
ARM="$1"; RUN="$2"; ROUNDS="${3:-7}"
|
||||
ROOT=/home/davide/Projects/SirRoboGarage
|
||||
OUT=/tmp/whichgun
|
||||
OUT="${WHICHGUN_OUT:-/tmp/whichgun}"
|
||||
BOTDIR="$OUT/drussgt_bots/${ARM}_${RUN}/DrussGT"
|
||||
mkdir -p "$BOTDIR" "$OUT/botlog"
|
||||
|
||||
|
||||
Reference in New Issue
Block a user