SirRoboGarage/tools/ab/README.md

tools/ab — reusable A/B harness

Two tools, built once and reused for every variant test. Adding an arm costs nothing: the frozen bot is built once per session and every arm reuses it.

1. Run a session

tools/ab/ab_run.sh --arms tools/ab/arms.example.txt --runs 7 --outdir /tmp/ab/power --conc 7

• builds ONE frozen ModularBot from current HEAD (git archive HEAD + nim c -d:release) and reuses that binary for every arm — a dirty tree cannot leak into the measurement;
• runs arm × run battles vs real DrussGT in parallel (ephemeral ports, one DrussGT botdir/data per run, one ModularBot botdir per run);
• writes session.json (commit, binary sha256, arms, runs/rounds, timestamp) and <outdir>/<arm>/run<N>.{jsonl,jsonl.rounds.json,jsonl.results.json,events.jsonl,battle.log,bot.stdout.log}.

Options: --arms FILE (required) --runs N (default 7) --outdir DIR (required) --conc K (default 7) --rounds R (default 7).

It kills its own children (own process group + outdir-tagged backstop) on EXIT/INT/TERM, so a Ctrl-C does not leave orphan battles.

Prerequisites (fails loudly if any is missing): /tmp/robocode/install/libs/robocode.jar, /tmp/drussgt/DrussGT.jar, the Tank Royale runner jar, the bot-API jar, nim, and the shim out/ classes. /tmp/tr_bots/DrussGT is recreated via make_botdir.sh if absent (the actual battles still use per-run copies).

2. Analyze a session

python3 tools/ab/ab_analyze.py /tmp/ab/power [--reference control]

Prints per-arm damage/run, damage taken/run, round wins, shots/run, hits taken/run, the per-run values, an exact two-sided permutation test on per-run damage and wins vs the reference arm (default: first arm), a round-level Fisher test (labelled anti-conservative), a liveness OK/FAIL line, and a round-win attribution cross-check.

Round wins come from the events sidecar (the bot that does not die wins) and are cross-checked against the runner's firstPlaces. The per-round lines in *.results.json are cumulative standings — not round winners.

Arm file

See arms.example.txt:

name | ENV_VAR=value ENV_VAR2=value2 | optional label

Known gotchas

• Ports: the runner picks ephemeral ports itself; nothing to configure.
• Races: never share a DrussGT botdir/data or a ModularBot stdout log across parallel runs — ab_run.sh already gives every run its own.
• pkill -f run_bridge_battle matches the pkill command itself; use the [r]un_bridge_battle trick (as ab_run.sh does).
• Liveness reads the bot's [env] boot report from <arm>/run<N>.bot.stdout.log; if an arm's variable is missing there it is a FAIL, not a measurement.