tools/ab: reusable A/B runner + analyzer (frozen-HEAD build, exact permutation test, liveness check)
This commit is contained in:
@@ -0,0 +1,65 @@
|
||||
# tools/ab — reusable A/B harness
|
||||
|
||||
Two tools, built once and reused for every variant test. Adding an arm costs
|
||||
nothing: the frozen bot is built once per session and every arm reuses it.
|
||||
|
||||
## 1. Run a session
|
||||
|
||||
```sh
|
||||
tools/ab/ab_run.sh --arms tools/ab/arms.example.txt --runs 7 --outdir /tmp/ab/power --conc 7
|
||||
```
|
||||
|
||||
* builds ONE frozen ModularBot from **current HEAD** (`git archive HEAD` +
|
||||
`nim c -d:release`) and reuses that binary for every arm — a dirty tree cannot
|
||||
leak into the measurement;
|
||||
* runs arm × run battles vs real DrussGT in parallel (ephemeral ports, one
|
||||
DrussGT botdir/data per run, one ModularBot botdir per run);
|
||||
* writes `session.json` (commit, binary sha256, arms, runs/rounds, timestamp)
|
||||
and `<outdir>/<arm>/run<N>.{jsonl,jsonl.rounds.json,jsonl.results.json,events.jsonl,battle.log,bot.stdout.log}`.
|
||||
|
||||
Options: `--arms FILE` (required) `--runs N` (default 7) `--outdir DIR`
|
||||
(required) `--conc K` (default 7) `--rounds R` (default 7).
|
||||
|
||||
It kills its own children (own process group + outdir-tagged backstop) on
|
||||
EXIT/INT/TERM, so a Ctrl-C does not leave orphan battles.
|
||||
|
||||
Prerequisites (fails loudly if any is missing):
|
||||
`/tmp/robocode/install/libs/robocode.jar`, `/tmp/drussgt/DrussGT.jar`,
|
||||
the Tank Royale runner jar, the bot-API jar, `nim`, and the shim `out/` classes.
|
||||
`/tmp/tr_bots/DrussGT` is recreated via `make_botdir.sh` if absent (the actual
|
||||
battles still use per-run copies).
|
||||
|
||||
## 2. Analyze a session
|
||||
|
||||
```sh
|
||||
python3 tools/ab/ab_analyze.py /tmp/ab/power [--reference control]
|
||||
```
|
||||
|
||||
Prints per-arm damage/run, damage taken/run, round wins, shots/run, hits
|
||||
taken/run, the **per-run** values, an exact two-sided permutation test on
|
||||
per-run damage and wins vs the reference arm (default: first arm), a
|
||||
round-level Fisher test (labelled anti-conservative), a liveness OK/FAIL line,
|
||||
and a round-win attribution cross-check.
|
||||
|
||||
Round wins come from the events sidecar (the bot that does not die wins) and
|
||||
are cross-checked against the runner's `firstPlaces`. The per-round lines in
|
||||
`*.results.json` are **cumulative** standings — not round winners.
|
||||
|
||||
## Arm file
|
||||
|
||||
See `arms.example.txt`:
|
||||
|
||||
```
|
||||
name | ENV_VAR=value ENV_VAR2=value2 | optional label
|
||||
```
|
||||
|
||||
## Known gotchas
|
||||
|
||||
* Ports: the runner picks ephemeral ports itself; nothing to configure.
|
||||
* Races: never share a DrussGT botdir/data or a ModularBot stdout log across
|
||||
parallel runs — `ab_run.sh` already gives every run its own.
|
||||
* `pkill -f run_bridge_battle` matches the pkill command itself; use the
|
||||
`[r]un_bridge_battle` trick (as `ab_run.sh` does).
|
||||
* Liveness reads the bot's `[env]` boot report from
|
||||
`<arm>/run<N>.bot.stdout.log`; if an arm's variable is missing there it is a
|
||||
FAIL, not a measurement.
|
||||
Executable
+520
@@ -0,0 +1,520 @@
|
||||
#!/usr/bin/env python3
|
||||
"""ab_analyze.py — read a session dir produced by ab_run.sh and print the report.
|
||||
|
||||
python3 tools/ab/ab_analyze.py <session_dir> [--reference ARM]
|
||||
|
||||
Standard library only, deterministic. For every arm it prints runs, damage/run,
|
||||
damage taken/run, ROUND WINS, shots/run, hits taken/run and the PER-RUN values
|
||||
(wins cluster at 0/7 and single-run damage swings ~200, so the mean alone lies).
|
||||
|
||||
Tests:
|
||||
* exact two-sided permutation test on PER-RUN values (damage/run and round
|
||||
wins) vs a reference arm (default: the first arm). C(14,7)=3432 for 7v7 —
|
||||
enumerated exactly, never sampled, whenever the combination count is small.
|
||||
* a round-level Fisher exact test on pooled rounds, clearly labelled
|
||||
anti-conservative (rounds cluster within runs).
|
||||
|
||||
Round-win attribution comes from the events sidecar (the bot that does NOT die
|
||||
wins the round) and is cross-checked against the runner's `firstPlaces`, which
|
||||
is name-based. `firstPlaces` is the run-level truth: the per-round lines in
|
||||
*.results.json are CUMULATIVE standings, not round winners — do not use them.
|
||||
|
||||
Liveness: each arm's declared env vars must appear verbatim in the bot's own
|
||||
boot environment report (<arm>/run<N>.bot.stdout.log, `[env] VAR=VALUE`), so an
|
||||
arm whose setting never reached the process is a loud FAIL rather than a
|
||||
plausible-looking number.
|
||||
|
||||
Owner attribution never relies on fired-power values (the power policy fires a
|
||||
continuous 0.15–1.15 range). The subject (DrussGT) is identified by matching
|
||||
the events sidecar's per-owner fire/hit counts against the capture's own
|
||||
`subject event counts:` line; ModularBot is the other bot.
|
||||
"""
|
||||
import glob
|
||||
import itertools
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
|
||||
# beyond this many combinations we sample (deterministic seed) and say so;
|
||||
# 7v7 = C(14,7) = 3432 is always exact.
|
||||
EXACT_CAP = 20_000_000
|
||||
BOT_NAME = "ModularBot"
|
||||
SUBJECT_NAME = "DrussGT" # the capture subject (our bot is the adversary)
|
||||
|
||||
|
||||
# ── loading ──────────────────────────────────────────────────────────────────
|
||||
|
||||
def load_session(outdir):
|
||||
p = os.path.join(outdir, "session.json")
|
||||
if os.path.exists(p):
|
||||
try:
|
||||
return json.load(open(p))
|
||||
except (json.JSONDecodeError, OSError):
|
||||
return None
|
||||
return None
|
||||
|
||||
|
||||
def parse_envspec(envspec):
|
||||
"""`TR_A=1 TR_B=2` -> {'TR_A': '1', 'TR_B': '2'}."""
|
||||
out = {}
|
||||
for tok in (envspec or "").split():
|
||||
if "=" in tok:
|
||||
k, v = tok.split("=", 1)
|
||||
out[k] = v
|
||||
return out
|
||||
|
||||
|
||||
def discover_arms(outdir, session):
|
||||
arms = []
|
||||
if session and isinstance(session.get("arms"), list):
|
||||
for a in session["arms"]:
|
||||
name = a.get("name")
|
||||
if not name:
|
||||
continue
|
||||
arms.append({"name": name,
|
||||
"env": parse_envspec(a.get("env", "")),
|
||||
"label": a.get("label", "")})
|
||||
if arms:
|
||||
return arms
|
||||
# fallback: any subdir holding run*.jsonl
|
||||
for d in sorted(os.listdir(outdir)):
|
||||
full = os.path.join(outdir, d)
|
||||
if os.path.isdir(full) and discover_runs(full):
|
||||
arms.append({"name": d, "env": {}, "label": ""})
|
||||
return arms
|
||||
|
||||
|
||||
def discover_runs(armdir):
|
||||
runs = []
|
||||
for f in os.listdir(armdir):
|
||||
m = re.fullmatch(r"run(\d+)\.jsonl", f)
|
||||
if m:
|
||||
runs.append(int(m.group(1)))
|
||||
return sorted(runs)
|
||||
|
||||
|
||||
def read_lines(path):
|
||||
try:
|
||||
with open(path, errors="replace") as fh:
|
||||
return fh.readlines()
|
||||
except OSError:
|
||||
return []
|
||||
|
||||
|
||||
def parse_events(path):
|
||||
evs = []
|
||||
for line in read_lines(path):
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
try:
|
||||
evs.append(json.loads(line))
|
||||
except json.JSONDecodeError:
|
||||
continue # partial line from an interrupted battle
|
||||
return evs
|
||||
|
||||
|
||||
def subject_counters(log_text):
|
||||
m = re.search(
|
||||
r"subject event counts: scans=(\d+) bulletsFired=(\d+) bulletHits=(\d+)"
|
||||
r" bulletMisses=(\d+) bulletHitBullets=(\d+) hitsTaken=(\d+)", log_text)
|
||||
if not m:
|
||||
return None
|
||||
return {"fired": int(m.group(2)), "hits": int(m.group(3)),
|
||||
"hits_taken": int(m.group(6))}
|
||||
|
||||
|
||||
def parse_round_wins_by_name(log_text):
|
||||
"""ModularBot's `firstPlaces` from the runner's final standings block."""
|
||||
for m in re.finditer(
|
||||
r"^\s*#\d+\s+(\S+)\s+totalScore=-?\d+\s+firstPlaces=(\d+)", log_text,
|
||||
re.MULTILINE):
|
||||
if m.group(1) == BOT_NAME:
|
||||
return int(m.group(2))
|
||||
return None
|
||||
|
||||
|
||||
def parse_round_scores(path):
|
||||
"""Per-round score delta per bot NAME from the cumulative *.results.json
|
||||
lines. Used only to break ties (mutual kill / no death), never as the
|
||||
primary attribution."""
|
||||
try:
|
||||
lines = json.load(open(path)).get("roundResults", [])
|
||||
except (OSError, json.JSONDecodeError, AttributeError):
|
||||
return {}
|
||||
prev, out = {}, {}
|
||||
for i, line in enumerate(lines, 1):
|
||||
cur = {m.group(1): int(m.group(2))
|
||||
for m in re.finditer(r"(\S+) rank=\d+ score=(-?\d+)", line)}
|
||||
if not cur:
|
||||
continue
|
||||
out[i] = {name: sc - prev.get(name, 0) for name, sc in cur.items()}
|
||||
prev = cur
|
||||
return out
|
||||
|
||||
|
||||
def attribute_owners(evs, counters):
|
||||
"""Return (subject_id, other_id) using fire/hit counts, never powers."""
|
||||
fires, hits, victim_hits = {}, {}, {}
|
||||
for o in evs:
|
||||
t = o.get("type")
|
||||
if t == "fire":
|
||||
fires[o["owner"]] = fires.get(o["owner"], 0) + 1
|
||||
elif t == "hit":
|
||||
hits[o["owner"]] = hits.get(o["owner"], 0) + 1
|
||||
if "victim" in o:
|
||||
victim_hits[o["victim"]] = victim_hits.get(o["victim"], 0) + 1
|
||||
if not fires:
|
||||
return None, None
|
||||
if counters:
|
||||
strict = [o for o in fires
|
||||
if fires[o] == counters["fired"]
|
||||
and hits.get(o, 0) == counters["hits"]
|
||||
and victim_hits.get(o, 0) == counters["hits_taken"]]
|
||||
if len(strict) == 1:
|
||||
subj = strict[0]
|
||||
return subj, _other(fires, subj)
|
||||
# relaxed fallbacks (logged by caller via `counters is None` etc.)
|
||||
cand = [o for o in fires if counters and fires[o] == counters["fired"]]
|
||||
if len(cand) != 1:
|
||||
cand = [o for o in fires
|
||||
if counters and victim_hits.get(o, 0) == counters["hits_taken"]]
|
||||
if len(cand) != 1:
|
||||
cand = list(fires)
|
||||
if len(cand) == 2:
|
||||
return cand[0], cand[1]
|
||||
if len(cand) == 1:
|
||||
return cand[0], _other(fires, cand[0])
|
||||
return None, None
|
||||
|
||||
|
||||
def _other(owners, subj):
|
||||
others = [o for o in owners if o != subj]
|
||||
return others[0] if len(others) == 1 else None
|
||||
|
||||
|
||||
def parse_run(armdir, run):
|
||||
"""Everything the report needs for one run; None-free on partial data."""
|
||||
evs = parse_events(os.path.join(armdir, f"run{run}.events.jsonl"))
|
||||
log_text = "".join(read_lines(os.path.join(armdir, f"run{run}.battle.log")))
|
||||
counters = subject_counters(log_text)
|
||||
subj, other = attribute_owners(evs, counters)
|
||||
|
||||
r = {"run": run, "rounds": 0, "wins": None, "first_places":
|
||||
parse_round_wins_by_name(log_text), "subject_id": subj,
|
||||
"other_id": other,
|
||||
"attribution_exact": counters is not None, "shots": 0, "damage": 0.0,
|
||||
"damage_taken": 0.0, "hits_taken": 0, "hits_dealt": 0,
|
||||
"death_solved": 0, "death_score_agree": 0, "ambiguous": 0}
|
||||
if subj is None or other is None:
|
||||
# still count rounds so the arm shows up, but flags will be set
|
||||
r["rounds"] = _round_count(armdir, run)
|
||||
return r
|
||||
|
||||
deaths = {}
|
||||
for o in evs:
|
||||
t = o.get("type")
|
||||
if t == "fire" and o.get("owner") == other:
|
||||
r["shots"] += 1
|
||||
elif t == "hit":
|
||||
if o.get("owner") == other:
|
||||
r["damage"] += o.get("damage", 0.0)
|
||||
r["hits_dealt"] += 1
|
||||
if o.get("victim") == other:
|
||||
r["damage_taken"] += o.get("damage", 0.0)
|
||||
r["hits_taken"] += 1
|
||||
elif t == "death":
|
||||
deaths.setdefault(o.get("round"), []).append(o.get("victim"))
|
||||
|
||||
nrounds = _round_count(armdir, run)
|
||||
r["rounds"] = nrounds
|
||||
scores = parse_round_scores(
|
||||
os.path.join(armdir, f"run{run}.jsonl.results.json"))
|
||||
# Per-round winner: primary = death events (a lone death means the OTHER bot
|
||||
# won). A round with two or zero deaths (a mutual kill or a timeout) cannot
|
||||
# be resolved from deaths alone, so the name-based per-round score delta is
|
||||
# the tie-break. The two methods are cross-checked on the unambiguous rounds.
|
||||
wins = death_solved = agree = ambiguous = 0
|
||||
for rd in range(1, nrounds + 1):
|
||||
vics = deaths.get(rd, [])
|
||||
winner = None
|
||||
if len(vics) == 1:
|
||||
winner = other if vics[0] == subj else subj
|
||||
death_solved += 1
|
||||
delta = scores.get(rd)
|
||||
if delta and SUBJECT_NAME in delta and BOT_NAME in delta:
|
||||
score_winner = (subj if delta[SUBJECT_NAME] > delta[BOT_NAME]
|
||||
else other)
|
||||
if winner is None:
|
||||
winner = score_winner
|
||||
ambiguous += 1
|
||||
elif winner == score_winner:
|
||||
agree += 1
|
||||
if winner == other:
|
||||
wins += 1
|
||||
r["wins"] = wins
|
||||
r["death_solved"] = death_solved
|
||||
r["death_score_agree"] = agree
|
||||
r["ambiguous"] = ambiguous
|
||||
return r
|
||||
|
||||
|
||||
def _round_count(armdir, run):
|
||||
p = os.path.join(armdir, f"run{run}.jsonl.rounds.json")
|
||||
try:
|
||||
return len(json.load(open(p)).get("rounds", []))
|
||||
except (OSError, json.JSONDecodeError, AttributeError):
|
||||
pass
|
||||
# fall back to distinct round numbers in the events sidecar
|
||||
rds = {o.get("round") for o in
|
||||
parse_events(os.path.join(armdir, f"run{run}.events.jsonl"))}
|
||||
return len(rds)
|
||||
|
||||
|
||||
# ── statistics ───────────────────────────────────────────────────────────────
|
||||
|
||||
def perm_test(xa, xb):
|
||||
"""Exact two-sided permutation test on the difference of means. Returns
|
||||
(obs, p, n_perm, exact_bool)."""
|
||||
na, nb = len(xa), len(xb)
|
||||
if na == 0 or nb == 0:
|
||||
return None
|
||||
obs = abs(sum(xa) / na - sum(xb) / nb)
|
||||
pooled = list(xa) + list(xb)
|
||||
n = na + nb
|
||||
total_sum = sum(pooled)
|
||||
ncomb = math.comb(n, na)
|
||||
if ncomb <= EXACT_CAP:
|
||||
cnt = 0
|
||||
for combo in itertools.combinations(range(n), na):
|
||||
sa = sum(pooled[i] for i in combo)
|
||||
if abs(sa / na - (total_sum - sa) / nb) >= obs - 1e-9:
|
||||
cnt += 1
|
||||
return obs, cnt / ncomb, ncomb, True
|
||||
# deterministic fallback for very large n (never hit at the default 7 runs)
|
||||
import random
|
||||
rng = random.Random(0xA1B2C3)
|
||||
B = 200_000
|
||||
cnt = 0
|
||||
for _ in range(B):
|
||||
idx = rng.sample(range(n), na)
|
||||
sa = sum(pooled[i] for i in idx)
|
||||
if abs(sa / na - (total_sum - sa) / nb) >= obs - 1e-9:
|
||||
cnt += 1
|
||||
return obs, (cnt + 1) / (B + 1), ncomb, False
|
||||
|
||||
|
||||
def fisher_two_sided(a, b, c, d):
|
||||
"""Two-sided Fisher exact on [[a,b],[c,d]]."""
|
||||
n = a + b + c + d
|
||||
if n == 0:
|
||||
return None
|
||||
row1, col1, col2 = a + b, a + c, b + d
|
||||
denom = math.comb(n, row1)
|
||||
if denom == 0:
|
||||
return None
|
||||
|
||||
def hyper(x):
|
||||
return math.comb(col1, x) * math.comb(col2, row1 - x) / denom
|
||||
|
||||
lo = max(0, col1 - (c + d))
|
||||
hi = min(row1, col1)
|
||||
obs = hyper(a)
|
||||
return sum(hyper(x) for x in range(lo, hi + 1) if hyper(x) <= obs + 1e-12)
|
||||
|
||||
|
||||
# ── liveness ─────────────────────────────────────────────────────────────────
|
||||
|
||||
def liveness(armdir, runs, env):
|
||||
"""OK iff every declared env var appears verbatim in the bot's boot report
|
||||
of every run, the report itself ran, and the bot did not warn that the name
|
||||
is unrecognised (an unknown TR_*/GUN_* var is silently ignored)."""
|
||||
if not runs:
|
||||
return "FAIL", "no runs"
|
||||
report_seen = {r: False for r in runs}
|
||||
matched = {r: True for r in runs}
|
||||
warned = {r: [] for r in runs}
|
||||
for r in runs:
|
||||
text = "".join(read_lines(os.path.join(
|
||||
armdir, f"run{r}.bot.stdout.log")))
|
||||
report_seen[r] = "=== ENVIRONMENT (boot report) ===" in text
|
||||
for k, v in env.items():
|
||||
if f"[env] {k}={v}" not in text:
|
||||
matched[r] = False
|
||||
if f"[env] WARNING: {k}" in text:
|
||||
warned[r].append(k)
|
||||
dead = [r for r in runs if not report_seen[r]]
|
||||
miss = [r for r in runs if not matched[r]]
|
||||
if dead:
|
||||
return "FAIL", f"no bot env report in run(s) {dead}"
|
||||
warn_runs = [r for r in runs if warned[r]]
|
||||
if warn_runs:
|
||||
names = sorted({k for r in warn_runs for k in warned[r]})
|
||||
return "FAIL", (f"bot ignored arm env {names} (unrecognised) in "
|
||||
f"run(s) {warn_runs}")
|
||||
if env and miss:
|
||||
vars_txt = " ".join(f"{k}={v}" for k, v in env.items())
|
||||
return "FAIL", f"arm env '{vars_txt}' not in boot report of run(s) {miss}"
|
||||
if not env:
|
||||
return "OK", f"{len(runs)}/{len(runs)} runs: no arm env; report present"
|
||||
vars_txt = " ".join(f"{k}={v}" for k, v in env.items())
|
||||
return "OK", f"{len(runs)}/{len(runs)} runs: {vars_txt} applied"
|
||||
|
||||
|
||||
# ── report ───────────────────────────────────────────────────────────────────
|
||||
|
||||
def main():
|
||||
args = sys.argv[1:]
|
||||
if not args or args[0] in ("-h", "--help"):
|
||||
print(__doc__)
|
||||
return 0 if args else 2
|
||||
outdir = args[0]
|
||||
reference = None
|
||||
if "--reference" in args:
|
||||
reference = args[args.index("--reference") + 1]
|
||||
if not os.path.isdir(outdir):
|
||||
print(f"ERROR: not a directory: {outdir}", file=sys.stderr)
|
||||
return 2
|
||||
|
||||
session = load_session(outdir)
|
||||
arms = discover_arms(outdir, session)
|
||||
if not arms:
|
||||
print(f"ERROR: no arms found in {outdir}", file=sys.stderr)
|
||||
return 2
|
||||
if reference is None:
|
||||
reference = arms[0]["name"]
|
||||
|
||||
data = {}
|
||||
for arm in arms:
|
||||
armdir = os.path.join(outdir, arm["name"])
|
||||
runs = discover_runs(armdir)
|
||||
per = [parse_run(armdir, r) for r in runs]
|
||||
data[arm["name"]] = {"dir": armdir, "runs": runs, "per": per,
|
||||
"env": arm["env"], "label": arm["label"]}
|
||||
|
||||
if session:
|
||||
print(f"# session {outdir}")
|
||||
print(f"# commit={session.get('commit','?')} "
|
||||
f"binary_sha256={session.get('binary_sha256','?')} "
|
||||
f"rounds={session.get('rounds','?')} runs={session.get('runs','?')} "
|
||||
f"conc={session.get('conc','?')} ts={session.get('timestamp','?')}")
|
||||
print()
|
||||
|
||||
# ── summary table ────────────────────────────────────────────────────────
|
||||
print("ARM SUMMARY")
|
||||
hdr = (f"{'arm':<14} {'runs':>4} {'dmg/run':>8} {'dmgtk/run':>9} "
|
||||
f"{'wins':>7} {'win%':>6} {'shots/run':>9} {'hitstk/run':>10}")
|
||||
print(hdr)
|
||||
print("-" * len(hdr))
|
||||
for arm in arms:
|
||||
name = arm["name"]
|
||||
per = data[name]["per"]
|
||||
n = len(per)
|
||||
if n == 0:
|
||||
print(f"{name:<14} {0:>4} (no runs)")
|
||||
continue
|
||||
dmg = sum(p["damage"] for p in per) / n
|
||||
dmgv = sum(p["damage_taken"] for p in per) / n
|
||||
shots = sum(p["shots"] for p in per) / n
|
||||
htk = sum(p["hits_taken"] for p in per) / n
|
||||
wins = sum(p["wins"] for p in per if p["wins"] is not None)
|
||||
rounds = sum(p["rounds"] for p in per)
|
||||
pct = f"{100.0 * wins / rounds:.1f}" if rounds else "n/a"
|
||||
print(f"{name:<14} {n:>4} {dmg:>8.0f} {dmgv:>9.0f} "
|
||||
f"{str(wins) + '/' + str(rounds):>7} {pct:>6} {shots:>9.0f} {htk:>10.1f}")
|
||||
|
||||
# ── per-run values ───────────────────────────────────────────────────────
|
||||
print("\nPER-RUN (never just the mean)")
|
||||
for arm in arms:
|
||||
name = arm["name"]
|
||||
per = data[name]["per"]
|
||||
dmgs = " ".join(
|
||||
f"r{p['run']}={p['damage']:.0f}" for p in per)
|
||||
wins = " ".join(
|
||||
f"r{p['run']}={p['wins']}/{p['rounds']}" if p["wins"] is not None
|
||||
else f"r{p['run']}=?/{p['rounds']}" for p in per)
|
||||
print(f" {name:<14} dmg: {dmgs}")
|
||||
print(f" {'':<14} wins: {wins}")
|
||||
|
||||
# ── exact permutation tests ──────────────────────────────────────────────
|
||||
if reference not in data:
|
||||
print(f"\nWARNING: reference arm '{reference}' not found; skipping tests")
|
||||
return 1
|
||||
print(f"\nEXACT TWO-SIDED PERMUTATION TEST (per-run values) vs `{reference}`")
|
||||
print(f"{'metric':<12} {'arm':<14} {'obs(diff)':>12} {'p':>8} permutations")
|
||||
print("-" * 64)
|
||||
ref = data[reference]["per"]
|
||||
for arm in arms:
|
||||
name = arm["name"]
|
||||
if name == reference:
|
||||
continue
|
||||
for label, key in (("dmg/run", "damage"), ("round wins", "wins")):
|
||||
xa = [p[key] for p in ref if p[key] is not None]
|
||||
xb = [p[key] for p in data[name]["per"] if p[key] is not None]
|
||||
res = perm_test(xa, xb)
|
||||
if res is None:
|
||||
print(f"{label:<12} {name:<14} {'n/a':>12} {'n/a':>8}")
|
||||
continue
|
||||
obs, p, ncomb, exact = res
|
||||
note = f"C({len(xa)+len(xb)},{len(xa)})={ncomb}" + (
|
||||
"" if exact else " SAMPLED")
|
||||
print(f"{label:<12} {name:<14} {obs:>+12.3f} {p:>8.4f} {note}")
|
||||
|
||||
# ── round-level test (anti-conservative) ─────────────────────────────────
|
||||
print("\nROUND-LEVEL TEST (pooled rounds, Fisher exact) vs "
|
||||
f"`{reference}` — ANTI-CONSERVATIVE: rounds cluster within runs")
|
||||
print(f"{'arm':<14} {'ref wins':>10} {'arm wins':>10} {'p':>8}")
|
||||
print("-" * 46)
|
||||
refw = sum(p["wins"] for p in ref if p["wins"] is not None)
|
||||
refr = sum(p["rounds"] for p in ref)
|
||||
for arm in arms:
|
||||
name = arm["name"]
|
||||
if name == reference:
|
||||
continue
|
||||
per = data[name]["per"]
|
||||
w = sum(p["wins"] for p in per if p["wins"] is not None)
|
||||
rr = sum(p["rounds"] for p in per)
|
||||
p = fisher_two_sided(w, rr - w, refw, refr - refw)
|
||||
pstr = f"{p:.4f}" if p is not None else "n/a"
|
||||
print(f"{name:<14} {str(refw) + '/' + str(refr):>10} "
|
||||
f"{str(w) + '/' + str(rr):>10} {pstr:>8}")
|
||||
|
||||
# ── liveness ─────────────────────────────────────────────────────────────
|
||||
print("\nLIVENESS (arm env applied in the bot's own boot report)")
|
||||
lv_fail = 0
|
||||
for arm in arms:
|
||||
name = arm["name"]
|
||||
status, why = liveness(data[name]["dir"], data[name]["runs"],
|
||||
data[name]["env"])
|
||||
if status == "FAIL":
|
||||
lv_fail += 1
|
||||
print(f" {name:<14} {status:<4} ({why})")
|
||||
|
||||
# ── round-win attribution cross-check ────────────────────────────────────
|
||||
print("\nROUND-WIN ATTRIBUTION (events primary; score tie-break for "
|
||||
"mutual-kill / timeout rounds)")
|
||||
for arm in arms:
|
||||
name = arm["name"]
|
||||
runs_ok = runs_tot = 0
|
||||
dsolve = dagree = damb = 0
|
||||
for p in data[name]["per"]:
|
||||
if p["wins"] is not None and p["first_places"] is not None:
|
||||
runs_tot += 1
|
||||
if p["wins"] == p["first_places"]:
|
||||
runs_ok += 1
|
||||
dsolve += p["death_solved"]
|
||||
dagree += p["death_score_agree"]
|
||||
damb += p["ambiguous"]
|
||||
status = ("OK" if runs_tot and runs_ok == runs_tot
|
||||
else ("n/a" if not runs_tot else "MISMATCH"))
|
||||
print(f" {name:<14} wins==firstPlaces {runs_ok}/{runs_tot} runs {status}; "
|
||||
f"single-death rounds agree with score {dagree}/{dsolve} "
|
||||
f"({damb} tie-broken)")
|
||||
|
||||
return 1 if lv_fail else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Executable
+270
@@ -0,0 +1,270 @@
|
||||
#!/usr/bin/env bash
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# ab_run.sh — run an A/B session of N arms vs real DrussGT.
|
||||
#
|
||||
# tools/ab/ab_run.sh --arms arms.txt --runs 7 --outdir /tmp/ab/foo --conc 7
|
||||
#
|
||||
# What it does, once per SESSION (not once per arm):
|
||||
# 1. builds ONE frozen ModularBot from the CURRENT HEAD via `git archive HEAD`
|
||||
# (so a dirty tree cannot leak into the measurement) and records the commit
|
||||
# sha + binary sha256;
|
||||
# 2. for every arm × run, launches a real battle against DrussGT in parallel,
|
||||
# each with its own ports (the runner picks ephemeral ones), its own
|
||||
# DrussGT botdir/data, its own ModularBot botdir and its own output files.
|
||||
#
|
||||
# Arm file format (one arm per line):
|
||||
# name | ENV_VAR=value ENV_VAR2=value2 | optional label
|
||||
# `name` alone or `name |` are also accepted (no arm env). Blank lines and
|
||||
# lines starting with `#` are ignored. See arms.example.txt.
|
||||
#
|
||||
# Output layout (what ab_analyze.py reads):
|
||||
# <outdir>/session.json commit, binary sha, arms, runs…
|
||||
# <outdir>/<arm>/run<N>.jsonl tick capture
|
||||
# <outdir>/<arm>/run<N>.jsonl.rounds.json round tick boundaries
|
||||
# <outdir>/<arm>/run<N>.jsonl.results.json
|
||||
# <outdir>/<arm>/run<N>.events.jsonl fire/hit/death sidecar
|
||||
# <outdir>/<arm>/run<N>.battle.log capture stdout (firstPlaces, …)
|
||||
# <outdir>/<arm>/run<N>.bot.stdout.log ModularBot boot env report (liveness)
|
||||
#
|
||||
# Cleanup: every job runs in its own process group (setsid); on EXIT/INT/TERM
|
||||
# this script SIGTERMs/SIGKILLs those groups and, as a backstop, pkills anything
|
||||
# whose command line references this session's outdir. So a Ctrl-C does not
|
||||
# leave orphan battles running.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
set -euo pipefail
|
||||
|
||||
REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)"
|
||||
RUN_BRIDGE="$REPO/tools/robocode_shim/run_bridge_battle.sh"
|
||||
MAKE_BOTDIR="$REPO/tools/robocode_shim/make_botdir.sh"
|
||||
|
||||
ARMS_FILE=""
|
||||
RUNS=7
|
||||
CONC=7
|
||||
OUTDIR=""
|
||||
ROUNDS=7
|
||||
|
||||
usage() {
|
||||
sed -n '2,40p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'
|
||||
exit "${1:-0}"
|
||||
}
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--arms) ARMS_FILE="$2"; shift 2;;
|
||||
--runs) RUNS="$2"; shift 2;;
|
||||
--outdir) OUTDIR="$2"; shift 2;;
|
||||
--conc) CONC="$2"; shift 2;;
|
||||
--rounds) ROUNDS="$2"; shift 2;;
|
||||
-h|--help) usage 0;;
|
||||
*) echo "unknown argument: $1" >&2; usage 1;;
|
||||
esac
|
||||
done
|
||||
|
||||
[[ -n "$ARMS_FILE" ]] || { echo "ERROR: --arms <file> is required" >&2; usage 1; }
|
||||
[[ -f "$ARMS_FILE" ]] || { echo "ERROR: arm file not found: $ARMS_FILE" >&2; exit 1; }
|
||||
[[ -n "$OUTDIR" ]] || { echo "ERROR: --outdir <dir> is required" >&2; usage 1; }
|
||||
[[ "$OUTDIR" != "/" && "$OUTDIR" != "" ]] || { echo "ERROR: refusing outdir '$OUTDIR'" >&2; exit 1; }
|
||||
OUTDIR="$(mkdir -p "$OUTDIR" && cd "$OUTDIR" && pwd)"
|
||||
|
||||
# ── prerequisites ────────────────────────────────────────────────────────────
|
||||
ROBOCODE_JAR="${ROBOCODE_JAR:-/tmp/robocode/install/libs/robocode.jar}"
|
||||
DRUSSGT_JAR="${DRUSSGT_JAR:-/tmp/drussgt/DrussGT.jar}"
|
||||
RUNNER_JAR="${TR_RUNNER_JAR:-/home/davide/Projects/tank-royale/runner/examples/lib/robocode-tankroyale-runner.jar}"
|
||||
BOT_API_JAR="${TR_BOT_API_JAR:-$HOME/Downloads/sample-bots-java-1.0.2/lib/robocode-tankroyale-bot-api-1.0.2.jar}"
|
||||
DRUSSGT_SHARED="${TR_DRUSSGT_BOTDIR:-/tmp/tr_bots/DrussGT}"
|
||||
|
||||
missing=0
|
||||
for f in "$ROBOCODE_JAR" "$DRUSSGT_JAR" "$RUNNER_JAR" "$BOT_API_JAR"; do
|
||||
if [[ ! -f "$f" ]]; then echo "ERROR: missing prerequisite: $f" >&2; missing=1; fi
|
||||
done
|
||||
command -v nim >/dev/null 2>&1 || { echo "ERROR: nim not on PATH" >&2; missing=1; }
|
||||
command -v git >/dev/null 2>&1 || { echo "ERROR: git not on PATH" >&2; missing=1; }
|
||||
[[ -f "$RUN_BRIDGE" ]] || { echo "ERROR: missing $RUN_BRIDGE" >&2; missing=1; }
|
||||
[[ -f "$MAKE_BOTDIR" ]] || { echo "ERROR: missing $MAKE_BOTDIR" >&2; missing=1; }
|
||||
(( missing == 0 )) || { echo "ERROR: prerequisites missing — not starting a broken session" >&2; exit 1; }
|
||||
|
||||
if [[ ! -d "$DRUSSGT_SHARED" ]]; then
|
||||
echo "[ab] recreating shared DrussGT botdir: $DRUSSGT_SHARED"
|
||||
"$MAKE_BOTDIR" "$DRUSSGT_SHARED" >/dev/null
|
||||
fi
|
||||
|
||||
# ── parse the arm file into parallel arrays ──────────────────────────────────
|
||||
trim() { local s="$1"; s="${s#"${s%%[![:space:]]*}"}"; s="${s%"${s##*[![:space:]]}"}"; printf '%s' "$s"; }
|
||||
|
||||
ARM_NAMES=(); ARM_ENVS=(); ARM_LABELS=()
|
||||
while IFS= read -r line || [[ -n "$line" ]]; do
|
||||
[[ -z "${line//[[:space:]]/}" ]] && continue
|
||||
[[ "$line" =~ ^[[:space:]]*# ]] && continue
|
||||
name="$(trim "${line%%|*}")"
|
||||
rest="${line#*|}"
|
||||
if [[ "$line" == *"|"* ]]; then
|
||||
envspec="$(trim "${rest%%|*}")"
|
||||
label="$(trim "${rest#*|}")"
|
||||
else
|
||||
envspec=""; label=""
|
||||
fi
|
||||
[[ -n "$name" ]] || { echo "ERROR: arm with empty name in $ARMS_FILE" >&2; exit 1; }
|
||||
ARM_NAMES+=("$name"); ARM_ENVS+=("$envspec"); ARM_LABELS+=("$label")
|
||||
done < "$ARMS_FILE"
|
||||
(( ${#ARM_NAMES[@]} > 0 )) || { echo "ERROR: no arms parsed from $ARMS_FILE" >&2; exit 1; }
|
||||
# duplicate names would silently overwrite each other's run files
|
||||
dupes="$(printf '%s\n' "${ARM_NAMES[@]}" | sort | uniq -d)"
|
||||
[[ -z "$dupes" ]] || { echo "ERROR: duplicate arm name(s): $dupes" >&2; exit 1; }
|
||||
|
||||
# ── fresh session dir ────────────────────────────────────────────────────────
|
||||
rm -rf "$OUTDIR"
|
||||
mkdir -p "$OUTDIR"
|
||||
WORK="$OUTDIR/.work"; mkdir -p "$WORK"
|
||||
|
||||
# ── build ONE frozen bot from HEAD ───────────────────────────────────────────
|
||||
COMMIT="$(git -C "$REPO" rev-parse HEAD)"
|
||||
FROZEN_DIR="$OUTDIR/frozen/ModularBot"
|
||||
FROZEN_BIN="$FROZEN_DIR/ModularBot_bin"
|
||||
FROZEN_JSON="$FROZEN_DIR/ModularBot.json"
|
||||
mkdir -p "$FROZEN_DIR"
|
||||
BUILDDIR="$WORK/head"
|
||||
mkdir -p "$BUILDDIR"
|
||||
echo "[ab] exporting HEAD ($COMMIT) -> $BUILDDIR"
|
||||
git -C "$REPO" archive HEAD | tar -x -C "$BUILDDIR"
|
||||
|
||||
echo "[ab] building frozen ModularBot (nim c -d:release)…"
|
||||
( cd "$BUILDDIR/ModularBot_garage" && \
|
||||
nim c -d:release --nimcache:"$OUTDIR/nimcache" --out:"$FROZEN_BIN" src/ModularBot.nim )
|
||||
[[ -x "$FROZEN_BIN" ]] || { echo "ERROR: frozen build failed" >&2; exit 1; }
|
||||
BINSHA="$(sha256sum "$FROZEN_BIN" | awk '{print $1}')"
|
||||
echo "[ab] frozen binary sha256=$BINSHA"
|
||||
|
||||
# canonical bot identity for every arm (the dir is /named/ ModularBot)
|
||||
cat > "$FROZEN_JSON" <<'JSON'
|
||||
{
|
||||
"name": "ModularBot",
|
||||
"version": "0.1.0",
|
||||
"authors": ["Davide Cappellini"],
|
||||
"description": "frozen A/B build of ModularBot (ab_run.sh)",
|
||||
"homepage": "",
|
||||
"countryCodes": ["IT"],
|
||||
"gameTypes": ["classic", "1v1"],
|
||||
"platform": "Nim",
|
||||
"programmingLang": "Nim"
|
||||
}
|
||||
JSON
|
||||
|
||||
# ── session.json ─────────────────────────────────────────────────────────────
|
||||
json_esc() { printf '%s' "$1" | sed 's/\\/\\\\/g; s/"/\\"/g'; }
|
||||
{
|
||||
printf '{\n'
|
||||
printf ' "commit": "%s",\n' "$COMMIT"
|
||||
printf ' "binary_sha256": "%s",\n' "$BINSHA"
|
||||
printf ' "binary": "frozen/ModularBot/ModularBot_bin",\n'
|
||||
printf ' "rounds": %d,\n' "$ROUNDS"
|
||||
printf ' "runs": %d,\n' "$RUNS"
|
||||
printf ' "conc": %d,\n' "$CONC"
|
||||
printf ' "timestamp": "%s",\n' "$(date -Is)"
|
||||
printf ' "outdir": "%s",\n' "$(json_esc "$OUTDIR")"
|
||||
printf ' "arms": [\n'
|
||||
for i in "${!ARM_NAMES[@]}"; do
|
||||
printf ' {"name": "%s", "env": "%s", "label": "%s"}' \
|
||||
"$(json_esc "${ARM_NAMES[$i]}")" "$(json_esc "${ARM_ENVS[$i]}")" "$(json_esc "${ARM_LABELS[$i]}")"
|
||||
(( i + 1 < ${#ARM_NAMES[@]} )) && printf ',' || true
|
||||
printf '\n'
|
||||
done
|
||||
printf ' ]\n}\n'
|
||||
} > "$OUTDIR/session.json"
|
||||
|
||||
# ── one job (run in its own process group) ───────────────────────────────────
|
||||
# exported so the detached `bash -c` subshell can call it; config via env
|
||||
export REPO RUN_BRIDGE ROUNDS OUTDIR WORK FROZEN_BIN FROZEN_JSON
|
||||
|
||||
ab_run_one() {
|
||||
local arm="$1" run="$2" envspec="$3"
|
||||
local dir="$OUTDIR/$arm"
|
||||
local work="$WORK/$arm/run$run"
|
||||
local botdir="$work/bots/ModularBot"
|
||||
mkdir -p "$dir" "$botdir" "$work/drussgt" "$work/data"
|
||||
|
||||
cp "$FROZEN_JSON" "$botdir/ModularBot.json"
|
||||
cat > "$botdir/ModularBot.sh" <<SH
|
||||
#!/bin/sh
|
||||
cd "\$(dirname "\$0")"
|
||||
exec ./ModularBot_bin > "$dir/run$run.bot.stdout.log" 2> "$dir/run$run.bot.stderr.log"
|
||||
SH
|
||||
chmod +x "$botdir/ModularBot.sh"
|
||||
ln -sf "$FROZEN_BIN" "$botdir/ModularBot_bin"
|
||||
|
||||
rm -f "$dir/run$run.jsonl" "$dir/run$run.jsonl.rounds.json" \
|
||||
"$dir/run$run.jsonl.results.json" "$dir/run$run.events.jsonl" \
|
||||
"$dir/run$run.status"
|
||||
|
||||
local rc=0
|
||||
# shellcheck disable=SC2086 # envspec is intentionally word-split
|
||||
env $envspec \
|
||||
DRUSSGT_BOTDIR="$work/drussgt/DrussGT" \
|
||||
DRUSSGT_DATA="$work/data" \
|
||||
AB_SESSION="$OUTDIR" \
|
||||
TR_EVENTS_OUT="$dir/run$run.events.jsonl" \
|
||||
timeout 900 "$RUN_BRIDGE" "$botdir" "$ROUNDS" "$dir/run$run.jsonl" \
|
||||
> "$dir/run$run.battle.log" 2>&1 || rc=$?
|
||||
echo "$rc" > "$dir/run$run.status"
|
||||
return 0
|
||||
}
|
||||
export -f ab_run_one
|
||||
|
||||
# ── cleanup: kill this session's children ────────────────────────────────────
|
||||
JOB_PIDS=()
|
||||
cleanup() {
|
||||
local rc=$?
|
||||
trap - EXIT INT TERM
|
||||
for p in "${JOB_PIDS[@]:-}"; do
|
||||
[[ -n "$p" ]] || continue
|
||||
kill -TERM -- "-$p" 2>/dev/null || kill -TERM "$p" 2>/dev/null || true
|
||||
done
|
||||
sleep 0.5
|
||||
for p in "${JOB_PIDS[@]:-}"; do
|
||||
[[ -n "$p" ]] || continue
|
||||
kill -KILL -- "-$p" 2>/dev/null || true
|
||||
done
|
||||
# backstop: any survivor whose cmdline references this session's outdir.
|
||||
# `[r]un_…` cannot match this script's own pkill command line.
|
||||
pkill -f "[r]un_bridge_battle.sh.*$OUTDIR" 2>/dev/null || true
|
||||
pkill -f "[r]obocode_shim.TrBattleCapture.*$OUTDIR" 2>/dev/null || true
|
||||
exit "$rc"
|
||||
}
|
||||
trap cleanup EXIT INT TERM
|
||||
|
||||
# ── launch the jobs, K at a time ─────────────────────────────────────────────
|
||||
echo "[ab] session: ${#ARM_NAMES[@]} arms × $RUNS runs, rounds=$ROUNDS, conc=$CONC"
|
||||
echo "[ab] outdir: $OUTDIR"
|
||||
RUNNING=0
|
||||
for ai in "${!ARM_NAMES[@]}"; do
|
||||
arm="${ARM_NAMES[$ai]}"; envspec="${ARM_ENVS[$ai]}"
|
||||
for (( r=1; r<=RUNS; r++ )); do
|
||||
setsid bash -c 'ab_run_one "$1" "$2" "$3"' _ "$arm" "$r" "$envspec" &
|
||||
JOB_PIDS+=("$!")
|
||||
RUNNING=$((RUNNING + 1))
|
||||
if (( RUNNING >= CONC )); then
|
||||
wait -n || true
|
||||
RUNNING=$((RUNNING - 1))
|
||||
fi
|
||||
done
|
||||
done
|
||||
wait || true
|
||||
|
||||
# jobs are done — do not let the EXIT trap signal reused PIDs
|
||||
JOB_PIDS=()
|
||||
|
||||
# ── aggregate status ─────────────────────────────────────────────────────────
|
||||
FAILED=0
|
||||
for ai in "${!ARM_NAMES[@]}"; do
|
||||
arm="${ARM_NAMES[$ai]}"
|
||||
for (( r=1; r<=RUNS; r++ )); do
|
||||
st="$(cat "$OUTDIR/$arm/run$r.status" 2>/dev/null || echo 999)"
|
||||
if [[ "$st" != "0" ]]; then
|
||||
echo "[ab] FAIL $arm run$r (rc=$st) — see $OUTDIR/$arm/run$r.battle.log" >&2
|
||||
FAILED=$((FAILED + 1))
|
||||
fi
|
||||
done
|
||||
done
|
||||
|
||||
echo "[ab] done: $(( ${#ARM_NAMES[@]} * RUNS - FAILED )) ok, $FAILED failed"
|
||||
echo "[ab] analyze with: python3 $REPO/tools/ab/ab_analyze.py $OUTDIR"
|
||||
(( FAILED == 0 )) || exit 1
|
||||
@@ -0,0 +1,15 @@
|
||||
# A/B arm file — one arm per line. Passed to ab_run.sh via --arms.
|
||||
#
|
||||
# Format:
|
||||
# name | ENV_VAR=value ENV_VAR2=value2 | optional label
|
||||
#
|
||||
# * `name` must be unique; it becomes <outdir>/<name>/.
|
||||
# * the env part is exported into the battle process (so the bot inherits it);
|
||||
# ab_analyze.py verifies each pair shows up in the bot's boot env report.
|
||||
# * the first arm is the default reference for the permutation test.
|
||||
# * `name` alone, or `name |`, is a control arm with no env override.
|
||||
# * blank lines and lines starting with `#` are ignored.
|
||||
|
||||
control | | unmodified power policy (TR_POWER_ENERGY_MIN default 0.5)
|
||||
pmin100 | TR_POWER_ENERGY_MIN=1.0 | never fire below power 1.0
|
||||
pmin075 | TR_POWER_ENERGY_MIN=0.75 | never fire below power 0.75
|
||||
Reference in New Issue
Block a user