SirRoboGarage/docs/gun_campaign.md

Gun campaign — ledger

Goal (owner's mandate, 2026-09-26 overnight): the movement campaign produced a replicated champion (TR_MOVEMENT=strafe, docs/movement_campaign.md) which a parallel job is shipping. "Continue until you found an amazing movement. When found do the same over for a gun." This file is the GUN campaign's single source of truth: later jobs append a ## Batch N section and never edit an earlier one (a wrong earlier number gets a correction line, not a rewrite). This file is the gun successor to docs/movement_campaign.md; read that first, then this.

---

0. The question

The shipped rack admits Pattern only (onlyPattern). That decision was taken because the virtual-fitness selector measured negative value at every rack size tested, and Pattern is the best single gun by the DrussGT hit-rate table (docs/selector_negative_value.md, docs/gun_rack_analysis.md, commit e0666a5). But every one of those measurements — like every pre-campaign movement claim — is DrussGT-heavy, and the movement campaign's standing lesson is that a one-opponent result is not a result.

So Batch 1 asks the direct question, on a frozen 15-opponent panel:

Does any rack gun, or any small gun configuration, beat the shipped Pattern on damage/run AND round wins across 15 opponents — or is onlyPattern CONFIRMED rather than merely assumed?

A null is a successful outcome here: it upgrades onlyPattern from an assumption to a measured verdict across the panel, which is itself valuable.

---

1. Protocol (identical to the movement campaign; the instrument is reused verbatim)

tools/ab/tournament_run.sh + tools/ab/tournament_analyze.py + tools/ab/panel_movement.txt (frozen panel), reused unchanged. The unit of evidence is the number of opponents, not the number of runs.

Element	Rule
Subject	ONE frozen binary built from git archive HEAD (tournament_run.sh records commit + binary sha256 in session.json)
Arms	env dicts only — no per-arm rebuild, ever; the arm file is a committed file (tools/ab/arms_gun_b1.txt)
Panel	the frozen tools/ab/panel_movement.txt (15 opponents). Adding/removing an opponent starts a new batch number
Pairing	per opponent: average the arm's runs, subtract the reference's average for that same opponent → one delta per opponent; then aggregate
Isolation	per-run bot dir + classic data dir, ephemeral ports, own process group; cleanup only by this session's outdir
Serialization	one battle fleet at a time via --wait-arena; never a broad pkill robocode_shim
Liveness	every declared env token must appear verbatim in OUR bot's own raw-env report, else the run is excluded and named
Never shipped	this is a measurement campaign: git status clean, shipped defaults untouched, .gitignore untouched

Movement is PINNED in every arm: TR_MOVEMENT=strafe

Reason (hard rule). A movement-default flip is landing from job j120 during this run, so the default engine can change underneath us. Every arm therefore sets TR_MOVEMENT=strafe explicitly. This (a) removes movement as a confound, (b) makes all arms share exactly one movement, so every delta is a pure gun delta, and (c) satisfies the analyzer's liveness rule for every arm (each declared token must appear verbatim; an undeclared leaked TR_MOVEMENT is fatal, but here every arm declares it). The reference arm is therefore the shipped gun rack under the pinned movement, not under the (mutable) default.

---

2. Pre-registered decision rules (fixed BEFORE Batch 1 ran)

1. Primary metrics: damage/run and ROUND WINS. Secondary/explanation only: damage taken/run, incoming hit rate, mean distance. Never hit rate alone — that trap has inverted six verdicts in this project.
2. BETTER than the reference iff one primary metric is up with a cross-opponent sign test p < 0.05 while the other does not go down; the mirror image for WORSE. Anything else is NOT DISTINGUISHABLE (a real answer, not a failure). Both the strict reading (the other metric's mean delta >= 0) and the substantive reading (not detectably down: not significant and smaller than that metric's MDE) are printed.
3. A verdict must survive the between-opponent spread: the pooled mean delta is reported with the SD across opponents, its SE, a 95% CI, and the MDE (α=0.05 two-sided, 80% power). An effect smaller than the MDE is reported as not detectable — never as absent, never as a win.
4. Somewhere to stop: if no arm beats the shipped pattern by rule 2 in Batch 1 and no arm shows a >= +MDE damage gain with p<0.10, then the gun stage's first phase is closed with "the shipped onlyPattern rack is the best gun configuration we have measured across the panel" — that is a successful outcome, and the campaign moves to a named next axis rather than inventing more arms. See What would make us stop.
5. No promotion off a single metric, a single opponent, or a single run. A change that wins damage by losing wins (or vice-versa) is not a win.
6. Every batch is shot with a pre-registered prediction stated in its section before the battles finish; a prediction that turns out wrong is recorded as wrong.

---

3. Stage 0 — what we already know (given, not re-derived)

fact	value	source
shipped rack	onlyPattern (id 5), all other guns off	common_libs/gun_harness/selector.nim
why	selector measured negative value at every rack size (16, lean8, lean6, pairPC/PK/PL) and 10/10 adversaries	docs/selector_negative_value.md, docs/gun_rack_analysis.md
Pattern hit rate vs DrussGT	~10.8%	given
KNN / Linear / Circular / WallBounce / GF vs DrussGT	5.6 / 3.0 / 2.9 / 2.7 / 2.1 %	given
BitBrain when idle	== Pattern (30 runs/arm: 97/210 vs 97/210 wins)	docs/bitbrain_vs_tmhorizon_ab.md
BitBrain across 32 legacy opponents	neutral: -3.6 dmg/run, +0.02 wins/run, p=0.53/0.91	docs/gauntlet_bitbrain_vs_pattern.md
lead amplitude	dead axis — full gain sweep {0, 0.25-1.0, 1.0, 1.25, 1.5, 2.0} run, nothing beats Pattern	docs/bitbrain_campaign.md
offline prediction checks	veto-only — may kill a design, never select one	docs/offline_harness_trust.md
spinner-specific claim	UNTESTED: no constant-turn spinner exists in the legacy roster (a note for a later job, not built here unless nearly free)	docs/gauntlet_bitbrain_vs_pattern.md

Movement context (why the pin): the movement champion is TR_MOVEMENT=strafe at its current defaults (range 325, tol 25, tilt 15/0.10); it replicated its win over the shipped tfil in four independent sessions (Batch 4: 52.6% vs 38.5% round-win rate). Movement is held constant at that champion for every gun arm.

---

Batch 1 — does anything beat the shipped Pattern?

Design. One frozen binary, six env-only arms, one frozen panel (tools/ab/panel_movement.txt, 15 opponents: 5 dodger, 3 pattern, 1 wall-follower, 1 corner-camper, 1 spinner, 1 rammer, 1 brawler, 2 aggressive megas), 3 runs × 3 rounds per (opponent, arm) = 270 battles. Arm file: tools/ab/arms_gun_b1.txt. Reference: pattern.

#	arm	env	what it isolates
1	pattern	TR_MOVEMENT=strafe	the arm to beat (shipped onlyPattern rack)
2	bitbrain	+ TR_RACK_PATTERN=off TR_RACK_BITBRAIN=both TR_BITBRAIN_GAINS=1.0,1.25,1.5,2.0 TR_BITBRAIN_MEM=decay	BitBrain-only, learned gain (panel replication of the 32-opp gauntlet at the pinned-movement standard)
3	tmhorizon	+ TR_RACK_PATTERN=off TR_RACK_TMHORIZON=both	TMHorizon-only corrector (predicted to lose; never panel-tested)
4	knn	+ TR_RACK_PATTERN=off TR_RACK_KNN=both	best non-Pattern single gun (never panel-tested)
5	rack_pk	+ TR_RACK_KNN=both	Pattern + KNN, selector ON (different-family hedge)
6	rack_pt	+ TR_RACK_TMHORIZON=both	Pattern + TMHorizon, selector ON (corrector-family hedge)

(+ = the pinned TR_MOVEMENT=strafe is present in every arm; see §1.)

Pre-registered prediction (written BEFORE the battles finished):

1. No arm beats pattern on BOTH primaries, and the point estimates sit inside the MDE. onlyPattern is confirmed across the panel.
2. bitbrain is a wash vs pattern (replicating the 32-opponent gauntlet: ±3.6 dmg/run, ≈0 wins) — the DrussGT-only gain-config penalty does not carry.
3. tmhorizon is WORSE than pattern (its own offline work predicts it needs ~80% side accuracy and can reach ~60%); no damage win, no win win.
4. knn is WORSE than pattern on damage (its DrussGT hit rate is roughly half Pattern's) and not better on wins.
5. The two small racks do not beat pattern; rack_pk lands closer to Pattern than knn-alone does (the selector at least partially hedges back), but the selector's poor ranking keeps it below the reference.
6. If any surprise exists, it is a rack arm on damage without wins — the ring-mover mirror-image trap — and it will not be read as a win.

Outcome — direct answer

MEASURED. Session /tmp/ab/j121_g1, commit 1d8143a15e4a038c39dfc2153ff518fa003b8612 (the pre-registration commit), frozen binary sha256 aa49a45fec20…, 15 opponents × 6 arms × 3 runs × 3 rounds = 270 battles, 0 failed, 0 never started, 0 liveness exclusions. Every arm declared TR_MOVEMENT=strafe, verified in each bot's own raw-env report; the reference rack line reads PATTERN for pattern and the overridden rack reads BITBRAIN / TMHORIZON / KNN for the others.

DIRECT ANSWER (MEASURED, n=15 opponents): NO — nothing beats the shipped Pattern across the panel on damage/run AND round wins. Every arm except knn is NOT DISTINGUISHABLE from pattern under both the strict and the substantive reading of the pre-registered rule. The best nominal challenger, tmhorizon, is +0.13 wins/run (MDE 0.36) and +8.7 dmg/run (MDE 13.8) — an effect ~3× smaller than the design can detect — and bitbrain is +0.11 wins / +10.3 dmg, also inside the MDE. knn is detectably WORSE on both primaries. The shipped onlyPattern rack is therefore CONFIRMED rather than merely assumed, at the resolution of this panel. That is a real result, not a failure.

Pooled dashboard (all valid runs — explanation only, NOT the verdict)

arm	runs	dmg/run	dmg taken/run	wins/run	round wins	win rate	incoming hit rate	mean distance
pattern (REF)	45	99.6	155.9	1.51	68/135	50.4%	12.84%	435
bitbrain	45	109.9	151.2	1.62	73/135	54.1%	12.87%	427
tmhorizon	45	108.3	153.7	1.64	74/135	54.8%	12.53%	428
knn	45	60.3	175.7	0.89	40/135	29.6%	13.31%	430
rack_pk	45	107.3	161.3	1.42	64/135	47.4%	13.65%	432
rack_pt	45	105.3	149.9	1.64	74/135	54.8%	12.90%	424

Cross-opponent aggregation (the verdict layer, damage + wins)

arm	metric	mean Δ	spread (SD)	95% CI	sign test (wins/n)	p(sign)	p(sign-flip)	Wilcoxon p	MDE
bitbrain	damage	+10.34	16.65	[+1.12, +19.57]	11/15	0.1185	0.03247	0.05006	12.05
bitbrain	wins	+0.11	0.48	[-0.16, +0.38]	6/9	0.5078	0.4922	0.4764	0.35
tmhorizon	damage	+8.71	19.10	[-1.86, +19.29]	11/15	0.1185	0.09918	0.1183	13.82
tmhorizon	wins	+0.13	0.50	[-0.14, +0.41]	5/9	1.0	0.4062	0.3118	0.36
knn	damage	−39.33	23.15	[−52.15, −26.51]	0/15	6.1e-5	6.1e-5	0.0007	16.75
knn	wins	−0.62	0.59	[−0.95, −0.30]	1/11	0.01172	0.001953	0.004948	0.43
rack_pk	damage	+7.68	32.09	[−10.10, +25.45]	9/15	0.6072	0.423	0.5895	23.21
rack_pk	wins	−0.09	0.60	[−0.42, +0.24]	6/11	1.0	0.6738	0.5932	0.43
rack_pt	damage	+5.69	16.50	[−3.45, +14.83]	10/15	0.3018	0.2026	0.2681	11.93
rack_pt	wins	+0.13	0.41	[−0.10, +0.36]	6/10	0.7539	0.3242	0.1997	0.30

The pre-registered verdict (verbatim)

rank	arm	Δwins/run	Δdmg/run	sign test wins	sign test dmg	verdict (strict)	verdict (substantive)
1	tmhorizon	+0.13	+8.7	5/9 p=1	11/15 p=0.1185	not distinguishable	not distinguishable
2	rack_pt	+0.13	+5.7	6/10 p=0.7539	10/15 p=0.3018	not distinguishable	not distinguishable
3	bitbrain	+0.11	+10.3	6/9 p=0.5078	11/15 p=0.1185	not distinguishable	not distinguishable
4	rack_pk	−0.09	+7.7	6/11 p=1	9/15 p=0.6072	not distinguishable	not distinguishable
5	knn	−0.62	−39.3	1/11 p=0.01172	0/15 p=6.104e-05	WORSE	not distinguishable

Reference pattern: 99.6 dmg/run, 1.51 wins/run, 12.84% incoming, 435 px.

Reading (MEASURED, with the mechanism)

• knn is a clean, large loss and is dropped. It deals −39 dmg/run and loses −0.62 wins/run, positive on 0/15 opponents on damage and 1/11 on wins (p=6e-5 / p=0.012). Its incoming hit rate is no worse (13.31% vs 12.84%) — it simply cannot aim: the same movement, a worse gun.
• The two small racks do not beat pattern. rack_pk (Pattern+KNN, selector ON) is not the average of Pattern and KNN: it recovers most of KNN's damage loss (+7.7 vs KNN-alone's −39.3) but still lands nominally below Pattern on wins (−0.09). rack_pt (Pattern+TMHorizon) is statistically indistinguishable from tmhorizon-alone — adding Pattern to the rack changes nothing, i.e. the selector picks the corrector essentially always. The selector's 16-gun negative verdict reproduces at a 2-gun rack: it does not manufacture a win it did not have.
• The only signal is a damage-side hint on the two Pattern-lead correctors (bitbrain +10.3, tmhorizon +8.7, both 11/15 opponents). It is below the MDE (12.0 / 13.8) and not significant by the pre-registered sign test (p=0.1185), so it is not a win and not a null. Note the sign-flip test on the mean is nominally significant for bitbrain (p=0.032) while the sign test is not — the positive deltas are larger than the negatives — but with 5 arms compared this is not compelling after multiplicity, and the effect is under the MDE. This is exactly the "damage without wins" pattern this project keeps paying for (the ring mover: +31 dmg/run and fewer wins); here the win deltas (+0.11/+0.13) are themselves sub-MDE.
• The reference is stable: pattern under TR_MOVEMENT=strafe measures 99.6 dmg/run and 50.4% round wins here, matching the movement campaign's strafe reference in Batch 4 (103.5 dmg/run, 52.6%) — same movement, same panel, different job.

Pre-registered predictions — scorecard (an honest count)

#	prediction	outcome
1	no arm beats pattern on both primaries, point estimates inside the MDE; onlyPattern confirmed	CORRECT
2	bitbrain is a wash vs pattern (gauntlet: −3.6 dmg, ≈0 wins)	CORRECT on the verdict, WRONG on the point estimate: +10.3 dmg here vs −3.6 in the 32-opponent gauntlet; both inside their MDEs. Direction differs, verdict (wash) holds
3	tmhorizon is WORSE than pattern (its own docs predict it loses)	WRONG: it is nominally better on both primaries (+8.7 dmg, +0.13 wins), though not distinguishable
4	knn is WORSE than pattern on damage, not better on wins	CORRECT (detectably on both)
5	the small racks do not beat pattern; rack_pk lands closer to Pattern than knn-alone	CORRECT
6	any surprise is a rack arm on damage without wins	PARTLY CORRECT: the nominal damage edge is on the single-gun correctors and the racks; every one of them is win-flat

---

Batch 2 — wider-panel calibration of the damage-side hint

The Batch-1 answer is decisive on the primary question (nothing beats pattern) but the two corrector arms carry a sub-MDE damage hint that rule 3 forbids calling either way. Resolving it needs more opponents, not more runs, so this batch re-runs exactly those two arms plus the reference on a wider panel.

Design. One frozen binary, three env-only arms, panel tools/ab/panel_gun_b2.txt = the j117 32-opponent legacy roster minus Aurora (inert in j117: one fire in 6 battles — it cannot reveal a gun regression) plus SpinBot = 33 opponents (a superset of Batch 1's 15). 3 runs × 3 rounds = 297 battles, --conc 6, --wait-arena. Arm file: tools/ab/arms_gun_b2.txt. Reference: pattern. Movement pinned TR_MOVEMENT=strafe in every arm (same rationale as Batch 1).

#	arm	env	role
1	pattern	TR_MOVEMENT=strafe	reference
2	bitbrain	+ TR_RACK_PATTERN=off TR_RACK_BITBRAIN=both TR_BITBRAIN_GAINS=1.0,1.25,1.5,2.0 TR_BITBRAIN_MEM=decay	largest Batch-1 damage edge (+10.3)
3	tmhorizon	+ TR_RACK_PATTERN=off TR_RACK_TMHORIZON=both	second Batch-1 damage edge (+8.7)

Pre-registered prediction (written BEFORE the battles finished):

1. The +10 dmg/run hint does not survive the wider panel: bitbrain and tmhorizon damage deltas fall toward zero and stay inside the new (smaller) MDE; the per-opponent sign test remains non-significant. Rationale: both arms come from the same Pattern-lead-correction base, and the prior 32-opponent gauntlet measured bitbrain at −3.6 dmg/run on a largely overlapping roster; +10 on 15 opponents is plausibly a small-panel fluctuation.
2. Round wins remain flat for both arms — if anything they regress toward zero.
3. If instead the damage hint survives at a significant cross-opponent sign test with wins not down, it is recorded as the first gun to beat pattern by rule 2 — and the campaign immediately looks for what the corrector is exploiting. That would be a genuine up-set, and it would not be spun as a null.

Outcome — Batch 2

(filled in by the Batch-2 results commit)

---

What to try next

(Rewritten after Batch 1 — these are recommendations, not results. Filled in the results commit below.)

What would make us stop

• Stop the first phase once a batch's best arm cannot beat pattern beyond the MDE, or when a gun arm's damage gain is bought with a detectable win loss. At that point "onlyPattern is the measured optimum of this rack" is the conclusion, not a failure (§2 rule 4).
• Stop a single batch early only for a contract violation (arena not free, liveness FAIL, non-zero exit rate) — never because the numbers look boring.
• Do not invent more arms on the same axis once two consecutive batches fail to improve on pattern beyond the MDE; move to a named new axis instead (candidate list in "What to try next").

---

How to run a batch (exact commands)

# 0. wait for the arena (j120's movement run may still be fighting)
TOURNAMENT_NIMCACHE=/tmp/nc_j121 \
tools/ab/tournament_run.sh \
    --arms    tools/ab/arms_gun_b1.txt \
    --panel   tools/ab/panel_movement.txt \
    --runs 3 --rounds 3 --conc 6 --wait-arena 45 \
    --reference pattern \
    --outdir /tmp/ab/j121_g1

# 1. the paired per-opponent table, sign tests, MDE and the pre-registered verdict
python3 tools/ab/tournament_analyze.py /tmp/ab/j121_g1 --reference pattern

--reference may be ANY arm of the session: re-analyzing an old session with a different reference is a free pairwise comparison with no battles.

Session log (outdirs are in /tmp and are NOT committed)

session	commit	battles	arms	verdict
/tmp/ab/j121_g1	(recorded at run time)	270	pattern, bitbrain, tmhorizon, knn, rack_pk, rack_pt	(Batch 1 outcome below)