SirStone
da4a971ca9
Adds common_libs/tests/measure_melee_bitbrain_ab.nim (+ .sh driver, .py analyzer,
committed per-run fixtures) and docs/melee_bitbrain_ab.md.
Experiment: 4-bot Free-For-All (ModularBot + WaveSurfer + PatternMover +
RandomMover), 4 arms x 16 runs x 7 rounds, frozen ModularBot from git archive
HEAD (commit 0f5cfe3, binary 11bba27), shipped tfil movement in every run.
Arms differ only in the gun rack: pattern (shipped), bb_round, bb_ret, bb_learn.
Result: NOT DETECTABLE. Score (server round score = damage + survival bonus)
differs by -63..+33 pts (perm p=0.16-0.71) against an MDE of 151 (~5.1%).
Every arm finishes rank 1. Round wins hint BitBrain's way (112/112 and 111/112
vs 109/112) but p=0.225 (MW 0.080), half the 0.40-win MDE.
Liveness proven: rack boot lines flip (rack active melee = PATTERN / BITBRAIN),
every run faced 3 distinct targets and ~66-69 target changes, and the bb arms
logged one [bb-reset] reason=target_change per switch. The melee premise was
exercised; the fast adaptation bought no measurable score edge at this sample.
65 lines
2.1 KiB
Bash
Executable File
65 lines
2.1 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# run_melee_bitbrain_ab.sh — drive the melee A/B (see measure_melee_bitbrain_ab.nim).
|
|
#
|
|
# Builds ONE frozen ModularBot (from `git archive HEAD`), then runs one process
|
|
# per arm in parallel; each process owns its own Tank Royale server on an
|
|
# ephemeral port. Output: <outdir>/<arm>/run<N>.json (+ .bot.stdout.log).
|
|
#
|
|
# common_libs/tests/run_melee_bitbrain_ab.sh [outdir]
|
|
#
|
|
# Env: MELEE_RUNS (default 16), MELEE_ROUNDS (default 7), MELEE_ARMS (space list).
|
|
# Kills its own process groups on EXIT/INT/TERM (setsid per arm).
|
|
set -euo pipefail
|
|
|
|
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
REPO="$(cd "$HERE/../.." && pwd)"
|
|
OUTDIR="${1:-/tmp/melee_bitbrain_ab}"
|
|
RUNS="${MELEE_RUNS:-16}"
|
|
ROUNDS="${MELEE_ROUNDS:-7}"
|
|
ARMS_STR="${MELEE_ARMS:-pattern bb_round bb_ret bb_learn}"
|
|
BIN="$HERE/measure_melee_bitbrain_ab"
|
|
|
|
[[ -x "$BIN" ]] || { echo "ERROR: build the runner first: nim c --path:common_libs $HERE/measure_melee_bitbrain_ab.nim" >&2; exit 1; }
|
|
mkdir -p "$OUTDIR"
|
|
|
|
echo "[driver] building frozen ModularBot…"
|
|
"$BIN" --outdir "$OUTDIR" --build-only
|
|
|
|
read -r -a ARMS <<< "$ARMS_STR"
|
|
PIDS=()
|
|
cleanup() {
|
|
local rc=$?
|
|
trap - EXIT INT TERM
|
|
for p in "${PIDS[@]:-}"; do
|
|
[[ -n "$p" ]] || continue
|
|
kill -TERM -- "-$p" 2>/dev/null || true
|
|
done
|
|
sleep 1
|
|
for p in "${PIDS[@]:-}"; do
|
|
[[ -n "$p" ]] || continue
|
|
kill -KILL -- "-$p" 2>/dev/null || true
|
|
done
|
|
# backstop: anything from this run's process group that outlived the runner
|
|
pkill -f "[m]easure_melee_bitbrain_ab.*$OUTDIR" 2>/dev/null || true
|
|
exit "$rc"
|
|
}
|
|
trap cleanup EXIT INT TERM
|
|
|
|
for arm in "${ARMS[@]}"; do
|
|
echo "[driver] launching arm $arm (runs=$RUNS rounds=$ROUNDS)"
|
|
setsid "$BIN" --outdir "$OUTDIR" --arm "$arm" \
|
|
--run-start 1 --run-count "$RUNS" --rounds "$ROUNDS" \
|
|
> "$OUTDIR/$arm.driver.log" 2>&1 &
|
|
PIDS+=("$!")
|
|
sleep 3 # stagger so concurrent servers do not race for the same free port
|
|
done
|
|
|
|
echo "[driver] all arms launched; waiting…"
|
|
FAIL=0
|
|
for p in "${PIDS[@]}"; do
|
|
wait "$p" || FAIL=$((FAIL + 1))
|
|
done
|
|
PIDS=()
|
|
echo "[driver] done (failed arm processes: $FAIL)"
|
|
exit "$FAIL"
|