SETTLED: no small rack beats Pattern alone; the selector is negative value on a GOOD rack
Follow-up toe0666a5, which showed Pattern alone (10.78%) beats the full rack (6.93%). That left two open questions: is a SMALL rack of good guns better than Pattern alone, and does the selector add value on a good rack (rather than only on the bloated one)? Both are now answered: NO and NO. 6 arms x 7 runs x 7 rounds, one frozen binary from CLEAN HEADe0666a5(built via `git archive`, source verified byte-identical to the clean tree), rack knobs only, 8 concurrent battles, real server-side hit rate vs the real DrussGT, exact two-sided permutation test on per-run rates. arm guns (selector active?) real % dmg/run p vs onlyPattern onlyPattern Pattern, NO selection 10.36 264 -- lean8 HeadOn,Linear,Circular,Accel,Pattern,GF,KNN,WallBounce 6.31 146 0.0169 lean6 lean8 - HeadOn 8.83 212 0.0262 pairPC Pattern + Circular 8.23 185 0.0460 pairPK Pattern + KNN 9.80 264 0.3998 pairPL Pattern + Linear 8.23 200 0.0035 The control replicates the prior run (10.36% vs 10.78% before; same binary tree, different build path). THE MECHANISM, from the per-arm selected-gun mix - the virtual signal keeps ranking the WRONG guns first, even on a two-gun rack: lean8: HeadOn 46.2% of ticks at 2.0% REAL; Pattern only 14.4% (12.6% real) lean6: Pattern 29.2% (10.4% real) vs KNN 25.3% (8.1%) and Linear 15.2% (8.0%) pairPC: Circular 66.8% (6.9% real) vs Pattern 33.2% (11.2% real) - over-picks Circular pairPL: Linear 57.7% (6.0% real) vs Pattern 42.3% (11.1% real) - over-picks Linear pairPK: Pattern 86.4% - ties ONLY because the selector happens to pick Pattern most of the time; it is numerically lower with identical dmg/run So the failure is NOT rack size. Pruning does not fix it; the ranking is wrong. VERDICT: ship `onlyPattern` - Pattern alone with selection bypassed - at 10.36% real and 264 dmg/run, vs lean8 6.31%/146 and the prior full rack 6.93%/159. This DIRECTLY CONTRADICTS the standing user directive to keep virtual-fitness selection, so it is recorded here plainly rather than quietly acted on: disable the selector (`TR_RACK_<every gun but PATTERN>=off`) pending a better fitness signal. The mechanism itself is left intact and functional so it can be re-enabled with one env var, and so it can be fixed rather than discarded. REMAINING CAVEAT: ONE ADVERSARY. All of this is vs DrussGT. Pattern as the default must be re-checked against other bots first - that is the next job. Extends the reusable harness (tools/ab/which_gun_arm_env.sh now has lean8/lean6/ pairPC/pairPK/pairPL; which_gun_analyze.py is parameterised by WHICHGUN_OUT and compares against both `full` and `onlyPattern`).
This commit is contained in:
@@ -7,8 +7,10 @@ Also reads gun_stats.jsonl to prove each arm was LIVE (selected-gun mix).
|
||||
"""
|
||||
import json, glob, os, re, sys, collections, itertools, math
|
||||
|
||||
OUT = "/tmp/whichgun"
|
||||
ARMS = ["full", "onlyLinear", "onlyPattern", "onlyGF", "onlyKNN"]
|
||||
OUT = os.environ.get("WHICHGUN_OUT", "/tmp/whichgun")
|
||||
ARMS = ["full", "onlyLinear", "onlyPattern", "onlyGF", "onlyKNN",
|
||||
"lean8", "lean6", "pairPC", "pairPK", "pairPL"]
|
||||
BASELINES = ["full", "onlyPattern"]
|
||||
MB_POWERS = {1.0, 1.5, 2.0, 3.0}
|
||||
|
||||
|
||||
@@ -40,7 +42,10 @@ def ev_stats(path, subject_fired):
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
o = json.loads(line)
|
||||
try:
|
||||
o = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue # partial line from an in-progress battle
|
||||
t = o.get("type")
|
||||
if t == "fire":
|
||||
fires[o["owner"]] += 1
|
||||
@@ -178,15 +183,17 @@ def main():
|
||||
pr = " ".join(f"r{p['run']}={p['rate']:.2f}" for p in res[a]["per"])
|
||||
print(f" {a:<12} {pr}")
|
||||
|
||||
if "full" in res and len(arms) > 1:
|
||||
for base in BASELINES:
|
||||
if base not in res or len(arms) < 2:
|
||||
continue
|
||||
print("\n" + "=" * 96)
|
||||
print("EXACT TWO-SIDED PERMUTATION TEST vs `full` (per-run rates)")
|
||||
print(f"EXACT TWO-SIDED PERMUTATION TEST vs `{base}` (per-run rates)")
|
||||
print("=" * 96)
|
||||
for a in arms:
|
||||
if a == "full":
|
||||
if a == base:
|
||||
continue
|
||||
obs, p = perm_test_exact(res["full"], res[a])
|
||||
print(f" full vs {a:<12}: mean-rate diff = {obs:+.2f} pp, exact two-sided p = {p:.4f}")
|
||||
obs, p = perm_test_exact(res[base], res[a])
|
||||
print(f" {base} vs {a:<12}: mean-rate diff = {obs:+.2f} pp, exact two-sided p = {p:.4f}")
|
||||
|
||||
print("\n" + "=" * 96)
|
||||
print("LIVENESS — selected-gun mix (sum of per-tick `selected` over all rounds)")
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
# emits `TR_RACK_<GUN>=off ...` for every gun except the one to keep.
|
||||
# usage: arm_env.sh <full|onlyLinear|onlyPattern|onlyGF|onlyKNN>
|
||||
# emits `TR_RACK_<GUN>=off ...` for every gun NOT in the arm.
|
||||
# usage: arm_env.sh <full|onlyLinear|onlyPattern|onlyGF|onlyKNN|lean8|lean6|pairPC|pairPK|pairPL>
|
||||
arm="$1"
|
||||
GUNS="HEADON LINEAR TSETLIN CIRCULAR GUESSFACTOR PATTERN WALLBOUNCE ACCEL STOPSHOT DISPLACE AVGLEAD DECAYGF KNN TMSELECT"
|
||||
case "$arm" in
|
||||
@@ -8,13 +8,19 @@ case "$arm" in
|
||||
onlyPattern) keep="PATTERN";;
|
||||
onlyGF) keep="GUESSFACTOR";;
|
||||
onlyKNN) keep="KNN";;
|
||||
# ── small-good-rack follow-up ──────────────────────────────────────────────
|
||||
lean8) keep="HEADON LINEAR CIRCULAR ACCEL PATTERN GUESSFACTOR KNN WALLBOUNCE";;
|
||||
lean6) keep="LINEAR CIRCULAR ACCEL PATTERN GUESSFACTOR KNN";;
|
||||
pairPC) keep="PATTERN CIRCULAR";;
|
||||
pairPK) keep="PATTERN KNN";;
|
||||
pairPL) keep="PATTERN LINEAR";;
|
||||
*) echo "unknown arm $arm" >&2; exit 1;;
|
||||
esac
|
||||
out=""
|
||||
if [ -n "$keep" ]; then
|
||||
for g in $GUNS; do
|
||||
[ "$g" = "$keep" ] && continue
|
||||
out="$out TR_RACK_${g}=off"
|
||||
done
|
||||
fi
|
||||
for g in $GUNS; do
|
||||
case " $keep " in
|
||||
*" $g "*) continue;;
|
||||
esac
|
||||
out="$out TR_RACK_${g}=off"
|
||||
done
|
||||
echo "$out"
|
||||
|
||||
Reference in New Issue
Block a user