The gun selector is NEGATIVE value: Pattern alone beats the full rack (p=0.0012)
MEASURED against the real DrussGT, one frozen binary built from clean HEAD, rack knobs only (no source edits), 5 arms x 7 runs x 7 rounds, 8 concurrent battles, judged ONLY on server-side real hit rate from the events sidecar, exact two-sided permutation test on per-run rates. arm runs shots hits real % dmg/run p vs full full (shipped) 7 3898 270 6.93 159 -- onlyPattern 7 4582 494 10.78 287 0.0012 <- BETTER onlyKNN 7 4033 207 5.13 119 0.1340 onlyLinear 7 3215 105 3.27 65 0.0082 onlyGF 7 3193 72 2.25 45 0.0012 Firing Pattern ALONE gives +3.85pp pooled hit rate and +80% damage per run, and it fires MORE shots (4582 vs 3898) - it dominates on rate and volume. This is not "any single gun wins" (full beats Linear, GF and KNN); it is specifically "Pattern alone beats the rack". WHY - the virtual fitness signal mis-ranks guns against real outcomes: - HeadOn is massively over-selected: 31.4% of ticks, the most real shots (1070), but only 4.5% REAL. It alone drags the rack down. - Pattern has the best virtual rank and near-best real rate (11.9%, rank 2), yet is selected only 22.6% of the time. - Linear's apparent strength was SELECTION BIAS: conditional on being selected it looked like 15.2% (n=33), but its UNCONDITIONAL rate (onlyLinear) is 3.27%. Every earlier per-gun "real rate" in this repo is conditional on selection and is therefore confounded. This experiment is the clean measurement. NOT YET SETTLED (do not overclaim): - ONE ADVERSARY. All of this is vs DrussGT. Pattern must be re-checked against other bots before it becomes the default on this evidence alone. - Whether a SMALL rack of good guns beats Pattern alone. The selector is negative value on the CURRENT bloated rack; that does not prove it is negative value on a rack of only good guns. That is the next experiment and it decides whether the selection apparatus is fixed or disabled. - The user's standing directive is to KEEP virtual-fitness selection. This measurement conflicts with it, so the next step tests the selector on a small good rack rather than assuming either answer. Context - three prior selection-side attempts all failed: hysteresis (7.02% -> 5.10%, p=0.002), commitment (7.17% -> 4.44%, p=0.0012), arrival-accuracy tie-break (7.08%, p=0.88 null). The per-tick random draw is load-bearing on three independent measurements. This experiment locates the real problem one level up: which guns are in the rack, and that the virtual signal ranks them wrongly. Preserves the reusable harness (tools/ab/which_gun_run_one.sh, which_gun_arm_env.sh, which_gun_analyze.py) and the full writeup (docs/selector_negative_value.md).
This commit is contained in:
@@ -0,0 +1,64 @@
|
||||
# The gun selector is currently NEGATIVE value
|
||||
|
||||
**Measured. The best single gun beats the full rack, and not by a little.**
|
||||
|
||||
| arm | runs | shots | hits | real % | dmg/run | per-run range | exact two-sided p vs `full` |
|
||||
|---|---:|---:|---:|---:|---:|---|---:|
|
||||
| `full` (shipped rack) | 7 | 3898 | 270 | **6.93** | 159 | 2.54–10.37 | — |
|
||||
| **`onlyPattern`** | 7 | 4582 | 494 | **10.78** | **287** | 9.14–11.85 | **0.0012** |
|
||||
| `onlyKNN` | 7 | 4033 | 207 | 5.13 | 119 | 4.15–6.47 | 0.1340 |
|
||||
| `onlyLinear` | 7 | 3215 | 105 | 3.27 | 65 | 1.85–4.69 | 0.0082 |
|
||||
| `onlyGF` | 7 | 3193 | 72 | 2.25 | 45 | 1.52–3.47 | 0.0012 |
|
||||
|
||||
**Firing `Pattern` alone: +3.85 pp pooled hit rate, +80% damage per run, p = 0.0012.**
|
||||
It also fires MORE shots (4582 vs 3898), so it dominates on rate *and* volume.
|
||||
|
||||
This is **not** "any single gun wins" — `full` beats `onlyLinear`, `onlyGF` and `onlyKNN`.
|
||||
It is specifically **"Pattern alone beats the rack"**.
|
||||
|
||||
## Method
|
||||
|
||||
One frozen binary built from clean `HEAD` via `git archive` (other agents had `common_libs/guns/*`
|
||||
dirty), sha256 `f02481d8…`. Arms selected with the rack knobs only (`TR_RACK_<GUN>=off`), no source
|
||||
edits. 5 arms × 7 runs × 7 rounds, 8 concurrent bridge battles, real DrussGT, judged ONLY on
|
||||
server-side real hit rate from the events sidecar. Exact two-sided permutation test on per-run rates
|
||||
(C(14,7)=3432 splits). Each arm's liveness verified from the selected-gun mix.
|
||||
|
||||
Harness preserved at `tools/ab/which_gun_*.sh` and `tools/ab/which_gun_analyze.py`.
|
||||
|
||||
## Why: the virtual fitness signal mis-ranks guns vs real outcomes
|
||||
|
||||
From the `full` arm's own selection mix and per-gun real rates:
|
||||
|
||||
- **`HeadOn` is massively over-selected** — **31.4% of ticks**, the most real shots (1070), but only
|
||||
**4.5% real**. It alone drags the rack down.
|
||||
- **`Pattern`** has the best *virtual* rank and near-best *real* rate (11.9%, real rank 2), yet is
|
||||
selected only **22.6%** of the time.
|
||||
- **`Linear`'s apparent strength was SELECTION BIAS.** Conditional on being selected it looked like
|
||||
15.2% (n=33); its *unconditional* rate (`onlyLinear`) is **3.27%**. Every earlier per-gun "real
|
||||
rate" in this repo is conditional on selection and is therefore confounded — this experiment is
|
||||
the clean measurement.
|
||||
|
||||
## What this does NOT yet settle
|
||||
|
||||
- **One adversary.** Everything here is vs DrussGT. `Pattern` should be re-checked against other
|
||||
bots before it becomes the default on the strength of this alone. (Supporting evidence: an offline
|
||||
audit found `Pattern` is the only gun competitive in *every* distance/speed bucket.)
|
||||
- **Whether a SMALL good rack beats `Pattern` alone.** The selector is negative value on the current
|
||||
bloated rack; that does not prove it is negative value on a rack of only good guns. That is the
|
||||
next experiment, and it decides whether the selection apparatus gets fixed or disabled.
|
||||
- **The user's directive was to KEEP the virtual-fitness selection mechanism.** This measurement
|
||||
conflicts with that directive, so the next step is to test the selector on a small, good rack
|
||||
rather than to assume either answer.
|
||||
|
||||
## Prior context: three failed selection-side attempts
|
||||
|
||||
| attempt | result |
|
||||
|---|---|
|
||||
| hysteresis (commit to incumbent) | 7.02% → 5.10%, p=0.002 |
|
||||
| commitment (remove the random draw) | 7.17% → 4.44%, p=0.0012 |
|
||||
| arrival-accuracy tie-break (rank by path, narrow by point) | 7.08%, p=0.88 — null |
|
||||
|
||||
So the per-tick random draw is load-bearing on three independent measurements, and no attempt to
|
||||
"smarten" the tied band has helped. This experiment shows the problem is one level up: **which guns
|
||||
are in the rack, and the fact that the virtual signal ranks them wrongly.**
|
||||
@@ -0,0 +1,207 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Analyze the which-gun A/B from server-side events sidecars.
|
||||
|
||||
Ground truth: real hit rate from the events sidecar (fire/hit, server-side).
|
||||
Per-run rates -> exact two-sided permutation test (enumerated when small).
|
||||
Also reads gun_stats.jsonl to prove each arm was LIVE (selected-gun mix).
|
||||
"""
|
||||
import json, glob, os, re, sys, collections, itertools, math
|
||||
|
||||
OUT = "/tmp/whichgun"
|
||||
ARMS = ["full", "onlyLinear", "onlyPattern", "onlyGF", "onlyKNN"]
|
||||
MB_POWERS = {1.0, 1.5, 2.0, 3.0}
|
||||
|
||||
|
||||
def runs_for(arm):
|
||||
rs = set()
|
||||
for f in glob.glob(f"{OUT}/events_{arm}_r*.json"):
|
||||
m = re.search(rf"_{re.escape(arm)}_r(\d+)\.json$", f)
|
||||
if m:
|
||||
rs.add(int(m.group(1)))
|
||||
return sorted(rs)
|
||||
|
||||
|
||||
def parse_subject_fired(logpath):
|
||||
if not os.path.exists(logpath):
|
||||
return None
|
||||
for line in open(logpath, errors="replace"):
|
||||
m = re.search(r"subject event counts:.*bulletsFired=(\d+)", line)
|
||||
if m:
|
||||
return int(m.group(1))
|
||||
return None
|
||||
|
||||
|
||||
def ev_stats(path, subject_fired):
|
||||
fires = collections.Counter()
|
||||
hits = collections.Counter()
|
||||
dmg = collections.Counter()
|
||||
powers = collections.defaultdict(set)
|
||||
for line in open(path):
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
o = json.loads(line)
|
||||
t = o.get("type")
|
||||
if t == "fire":
|
||||
fires[o["owner"]] += 1
|
||||
powers[o["owner"]].add(round(o.get("power", -1), 3))
|
||||
elif t == "hit":
|
||||
hits[o["owner"]] += 1
|
||||
dmg[o["owner"]] += o.get("damage", 0.0)
|
||||
# ModularBot = owner whose power set is a subset of the discrete bins
|
||||
mb = None
|
||||
for own, pw in powers.items():
|
||||
if fires[own] > 0 and pw <= MB_POWERS:
|
||||
mb = own
|
||||
if mb is None:
|
||||
# fallback: owner that is NOT the subject (identified by bulletsFired)
|
||||
subj = None
|
||||
if subject_fired is not None:
|
||||
for own, n in fires.items():
|
||||
if n == subject_fired:
|
||||
subj = own
|
||||
cand = [o for o in fires if o != subj]
|
||||
if len(cand) == 1:
|
||||
mb = cand[0]
|
||||
if mb is None:
|
||||
return None
|
||||
return fires[mb], hits[mb], dmg[mb]
|
||||
|
||||
|
||||
def gun_mix(arm):
|
||||
"""Sum selected/realShots/realHits per gun across all rounds of the arm."""
|
||||
mix = collections.defaultdict(lambda: collections.Counter())
|
||||
rounds = 0
|
||||
for r in runs_for(arm):
|
||||
p = f"{OUT}/gun_stats_{arm}_r{r}.jsonl"
|
||||
if not os.path.exists(p):
|
||||
continue
|
||||
for line in open(p):
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
o = json.loads(line)
|
||||
rounds += 1
|
||||
for g in o["guns"]:
|
||||
a = mix[g["name"]]
|
||||
a["selected"] += g["selected"]
|
||||
a["realShots"] += g["realShots"]
|
||||
a["realHits"] += g["realHits"]
|
||||
return mix, rounds
|
||||
|
||||
|
||||
def analyze(arm, verbose=False):
|
||||
per = []
|
||||
totF = totH = 0
|
||||
totD = 0.0
|
||||
for r in runs_for(arm):
|
||||
ev = ev_stats(f"{OUT}/events_{arm}_r{r}.json", parse_subject_fired(f"{OUT}/battle_{arm}_r{r}.log"))
|
||||
if not ev:
|
||||
if verbose:
|
||||
print(f" r{r}: no events")
|
||||
continue
|
||||
f, h, d = ev
|
||||
rate = 100.0 * h / max(1, f)
|
||||
per.append({"run": r, "fires": f, "hits": h, "rate": rate, "dmg": d})
|
||||
totF += f
|
||||
totH += h
|
||||
totD += d
|
||||
if verbose:
|
||||
print(f" r{r}: {h}/{f} = {rate:.2f}% dmg={d:.0f}")
|
||||
return dict(arm=arm, per=per, shots=totF, hits=totH, dmg=totD,
|
||||
rate=100.0 * totH / max(1, totF), n=len(per))
|
||||
|
||||
|
||||
def perm_test_exact(a, b):
|
||||
"""Exact two-sided permutation test on per-run rates (difference of means)."""
|
||||
xa = [p["rate"] for p in a["per"]]
|
||||
xb = [p["rate"] for p in b["per"]]
|
||||
na, nb = len(xa), len(xb)
|
||||
if na == 0 or nb == 0:
|
||||
return float("nan"), float("nan")
|
||||
obs = abs(sum(xa) / na - sum(xb) / nb)
|
||||
pooled = xa + xb
|
||||
n = na + nb
|
||||
cnt = 0
|
||||
total = 0
|
||||
# exact enumeration of C(n, na); fall back to sampling for large n
|
||||
if math.comb(n, na) <= 200000:
|
||||
for combo in itertools.combinations(range(n), na):
|
||||
s = set(combo)
|
||||
ma = sum(pooled[i] for i in combo) / na
|
||||
mb = sum(pooled[i] for i in range(n) if i not in s) / nb
|
||||
if abs(ma - mb) >= obs - 1e-12:
|
||||
cnt += 1
|
||||
total += 1
|
||||
return obs, cnt / total
|
||||
# sampled
|
||||
import random
|
||||
rng = random.Random(12345)
|
||||
B = 200000
|
||||
for _ in range(B):
|
||||
idx = rng.sample(range(n), na)
|
||||
s = set(idx)
|
||||
ma = sum(pooled[i] for i in idx) / na
|
||||
mb = sum(pooled[i] for i in range(n) if i not in s) / nb
|
||||
if abs(ma - mb) >= obs - 1e-12:
|
||||
cnt += 1
|
||||
return obs, (cnt + 1) / (B + 1)
|
||||
|
||||
|
||||
def fmt_range(per):
|
||||
if not per:
|
||||
return "n/a"
|
||||
rs = [p["rate"] for p in per]
|
||||
return f"{min(rs):.2f}-{max(rs):.2f}"
|
||||
|
||||
|
||||
def main():
|
||||
arms = [a for a in ARMS if runs_for(a)]
|
||||
extra = sys.argv[1:]
|
||||
if extra:
|
||||
arms = extra
|
||||
res = {a: analyze(a, verbose=("-v" in sys.argv)) for a in arms}
|
||||
|
||||
print("=" * 96)
|
||||
print("REAL HIT RATE (server-side events sidecar) — ModularBot vs real DrussGT")
|
||||
print("=" * 96)
|
||||
hdr = f"{'arm':<12} {'runs':>4} {'shots':>6} {'hits':>5} {'real %':>7} {'dmg/run':>8} {'per-run range':>14}"
|
||||
print(hdr)
|
||||
print("-" * len(hdr))
|
||||
for a in arms:
|
||||
r = res[a]
|
||||
print(f"{a:<12} {r['n']:>4} {r['shots']:>6} {r['hits']:>5} {r['rate']:>7.2f} "
|
||||
f"{r['dmg']/max(1,r['n']):>8.0f} {fmt_range(r['per']):>14}")
|
||||
|
||||
print("\nper-run rates:")
|
||||
for a in arms:
|
||||
pr = " ".join(f"r{p['run']}={p['rate']:.2f}" for p in res[a]["per"])
|
||||
print(f" {a:<12} {pr}")
|
||||
|
||||
if "full" in res and len(arms) > 1:
|
||||
print("\n" + "=" * 96)
|
||||
print("EXACT TWO-SIDED PERMUTATION TEST vs `full` (per-run rates)")
|
||||
print("=" * 96)
|
||||
for a in arms:
|
||||
if a == "full":
|
||||
continue
|
||||
obs, p = perm_test_exact(res["full"], res[a])
|
||||
print(f" full vs {a:<12}: mean-rate diff = {obs:+.2f} pp, exact two-sided p = {p:.4f}")
|
||||
|
||||
print("\n" + "=" * 96)
|
||||
print("LIVENESS — selected-gun mix (sum of per-tick `selected` over all rounds)")
|
||||
print("=" * 96)
|
||||
for a in arms:
|
||||
mix, rounds = gun_mix(a)
|
||||
tot_sel = sum(v["selected"] for v in mix.values())
|
||||
parts = []
|
||||
for name, v in sorted(mix.items(), key=lambda kv: -kv[1]["selected"]):
|
||||
if v["selected"] == 0:
|
||||
continue
|
||||
share = 100.0 * v["selected"] / max(1, tot_sel)
|
||||
parts.append(f"{name} {share:.1f}% (sel={v['selected']}, real={v['realHits']}/{v['realShots']})")
|
||||
print(f" {a:<12} rounds={rounds:>3} " + ("; ".join(parts) if parts else "NO SELECTION RECORDED"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+20
@@ -0,0 +1,20 @@
|
||||
# emits `TR_RACK_<GUN>=off ...` for every gun except the one to keep.
|
||||
# usage: arm_env.sh <full|onlyLinear|onlyPattern|onlyGF|onlyKNN>
|
||||
arm="$1"
|
||||
GUNS="HEADON LINEAR TSETLIN CIRCULAR GUESSFACTOR PATTERN WALLBOUNCE ACCEL STOPSHOT DISPLACE AVGLEAD DECAYGF KNN TMSELECT"
|
||||
case "$arm" in
|
||||
full) keep="";;
|
||||
onlyLinear) keep="LINEAR";;
|
||||
onlyPattern) keep="PATTERN";;
|
||||
onlyGF) keep="GUESSFACTOR";;
|
||||
onlyKNN) keep="KNN";;
|
||||
*) echo "unknown arm $arm" >&2; exit 1;;
|
||||
esac
|
||||
out=""
|
||||
if [ -n "$keep" ]; then
|
||||
for g in $GUNS; do
|
||||
[ "$g" = "$keep" ] && continue
|
||||
out="$out TR_RACK_${g}=off"
|
||||
done
|
||||
fi
|
||||
echo "$out"
|
||||
Executable
+27
@@ -0,0 +1,27 @@
|
||||
#!/usr/bin/env bash
|
||||
# Run ONE bridge battle for one arm/run against the frozen binary.
|
||||
# run_one.sh <arm> <run> <rounds>
|
||||
set -u
|
||||
ARM="$1"; RUN="$2"; ROUNDS="${3:-7}"
|
||||
ROOT=/home/davide/Projects/SirRoboGarage
|
||||
OUT=/tmp/whichgun
|
||||
BOTDIR="$OUT/drussgt_bots/${ARM}_${RUN}/DrussGT"
|
||||
mkdir -p "$BOTDIR" "$OUT/botlog"
|
||||
|
||||
# arm env knobs (empty for `full`)
|
||||
# shellcheck disable=SC2046
|
||||
ARMV=$($OUT/arm_env.sh "$ARM")
|
||||
|
||||
rm -f "$OUT/gun_stats_${ARM}_r${RUN}.jsonl" "$OUT/events_${ARM}_r${RUN}.json" "$OUT/cap_${ARM}_r${RUN}.jsonl"
|
||||
|
||||
# shellcheck disable=SC2086
|
||||
env $ARMV \
|
||||
DRUSSGT_BOTDIR="$BOTDIR" \
|
||||
DRUSSGT_DATA="$OUT/drussgt_data/${ARM}_${RUN}" \
|
||||
GUN_STATS_PATH="$OUT/gun_stats_${ARM}_r${RUN}.jsonl" \
|
||||
TR_EVENTS_OUT="$OUT/events_${ARM}_r${RUN}.json" \
|
||||
WHICHGUN_LOG="$OUT/botlog/${ARM}_r${RUN}.log" \
|
||||
timeout 600 "$ROOT/tools/robocode_shim/run_bridge_battle.sh" \
|
||||
"$OUT/bots/ModularBot" "$ROUNDS" "$OUT/cap_${ARM}_r${RUN}.jsonl" \
|
||||
> "$OUT/battle_${ARM}_r${RUN}.log" 2>&1
|
||||
echo "$? ${ARM} r${RUN}"
|
||||
Reference in New Issue
Block a user