SirStone
c034eb9d25
- fix(ModularBot): onBulletHitBot → onBulletHit (real hits were never tracked) - feat(ModularBot): per-round gun stats dump to /tmp/gun_stats.jsonl - feat(ModularBot): gun selection counter per round - fix(tests): adversary paths _garage suffix removed from 7 test files - feat(tests): test_gauntlet_5bots.nim — 10-round gauntlet vs all 5 adversaries - feat(tests): analyze_gun_stats.nim — JSONL parser for gun performance tables - docs: gun_rack_analysis.md — full per-gun performance report - docs: gun_rack_summary.md — TL;DR verdict table (keep/drop/tune) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
16 KiB
Gun Rack Analysis Report
Date: 2026-09-20
Bot: ModularBot (14 guns)
Data source: /tmp/gun_stats.jsonl
Test Methodology
- Adversaries: 5 bots, 10 rounds each (OscillatorBot: 9 rounds — one round lost to disconnect)
- Format: 1v1, score accumulated across rounds
- Virtual bullet system: 4 power bins (1.0, 1.5, 2.0, 3.0), 100-tick rolling window per gun per bin
- Gun selection rule: highest virtual hit rate, minimum 15 observations before a gun is eligible to compete
- Stats written to:
/tmp/gun_stats.jsonl(one JSON line per round, prefixed bysession_startmarkers)
Adversary Profiles
| Adversary | Movement | What it tests |
|---|---|---|
| SittingDuck | Stationary | Baseline ceiling — any gun should near-100%; exposes broken guns |
| OscillatorBot | Periodic side-to-side oscillation | Predictable but timed — rewards guns that track phase, punishes naive straight-line |
| RandomMover | Uniform random heading changes | Stochastic coverage — tests gun generality under noise |
| PatternMover | Repeating movement macro | Tests pattern/memory guns; predictable once pattern is locked |
| WaveSurfer | Surfing (dodges incoming bullets reactively) | Most realistic; tests guns that account for dodge response |
Results Summary
| Adversary | ModularBot Score | Real Hits / Shots | Real Hit% |
|---|---|---|---|
| SittingDuck | 1944 | 69 / 81 | 85% |
| OscillatorBot | 1645 | 100 / 170 | 59% |
| RandomMover | 1916 | 86 / 142 | 61% |
| PatternMover | 1914 | 80 / 109 | 73% |
| WaveSurfer | 1873 | 76 / 105 | 72% |
Note: Real hit% was dead during these tests due to the
onBulletHitBotbug (see Bugs section). Numbers reflect post-fix tracking where the callback fired correctly, but per-gun real-hit attribution was not collected — only aggregate real hits per round are reliable.
Per-Gun Virtual Hit Rate Matrix
Average virtual hit rate (%) across all rounds per adversary. Bold = best gun for that adversary. Selection count (total ticks gun was active) shown in parentheses.
| Gun | SittingDuck | OscillatorBot | RandomMover | PatternMover | WaveSurfer |
|---|---|---|---|---|---|
| HeadOn | 98% (1470) | 63% (664) | 79% (1010) | 95% (1188) | 84% (1215) |
| Linear | 98% (0) | 63% (552) | 82% (251) | 95% (27) | 84% (13) |
| Tsetlin | 98% (0) | 63% (0) | 82% (0) | 95% (0) | 84% (0) |
| Circular | 98% (0) | 63% (3) | 80% (350) | 95% (70) | 84% (137) |
| GuessFactor | 98% (0) | 63% (0) | 79% (100) | 94% (0) | 84% (0) |
| Pattern | 98% (0) | 9% (1081) | 17% (254) | 95% (0) | 84% (77) |
| AntiSurf | 0% (0) | 0% (147) | 2% (0) | 0% (19) | 0% (0) |
| WallBounce | 99% (68) | 64% (82) | 82% (215) | 95% (211) | 85% (51) |
| Accel | 99% (0) | 63% (281) | 19% (152) | 96% (24) | 86% (255) |
| StopShot | 99% (0) | 64% (1) | 82% (1) | 96% (6) | 86% (1) |
| Displace | 99% (0) | 63% (5) | 81% (55) | 96% (243) | 86% (10) |
| AvgLead | 99% (0) | 65% (105) | 82% (110) | 96% (107) | 86% (97) |
| DecayGF | 99% (0) | 63% (0) | 79% (1) | 96% (20) | 86% (30) |
| KNN | 99% (0) | 63% (0) | 79% (1) | 96% (50) | 85% (24) |
Best gun per adversary:
- SittingDuck → WallBounce / Accel / StopShot / Displace / AvgLead / DecayGF / KNN (all tied at 99%)
- OscillatorBot → AvgLead (65%)
- RandomMover → Linear / Tsetlin / WallBounce / StopShot / AvgLead (tied at 82%)
- PatternMover → Displace / Accel / StopShot / DecayGF / AvgLead / KNN (tied at 96%)
- WaveSurfer → StopShot / DecayGF / AvgLead / Accel / Displace (tied at 86%)
Gun-by-Gun Analysis
HeadOn
What it does: Fires directly at current enemy position — no lead, no prediction.
- Strengths: SittingDuck (98%), PatternMover (95%), WaveSurfer (84%). Competent baseline everywhere.
- Weaknesses: No lead — loses to faster movers once Random/Oscillator velocity matters. Matched or beaten by 8+ other guns on every adversary except SittingDuck.
- Selection bias: Receives the most ticks in every matchup (664–1470), consistently crowding out marginally better guns. This is a selector problem, not a gun problem.
- Verdict: KEEP — but its selection dominance needs addressing. It is a reliable floor, not the ceiling.
Linear
What it does: Straight-line lead — assumes constant velocity from last observed heading.
- Strengths: RandomMover (82%, tied best), WaveSurfer (84%), PatternMover (95%).
- Weaknesses: OscillatorBot (63%, same as HeadOn) — oscillation breaks the constant-velocity assumption.
- Verdict: KEEP — genuinely better than HeadOn on random movers. Gets 0 selection ticks on most matchups despite competitive rates; underutilised.
Tsetlin
What it does: Tsetlin Machine classifier for aim prediction (binary learning automaton).
- Strengths: Identical rates to Linear across all adversaries (82% Random, 63% Oscillator, 95% Pattern, 84% Wave, 98% Sitting).
- Weaknesses: Gets 0 selection ticks in every single matchup — never selected once. Either never crosses the 15-observation threshold, or its virtual bullets are being computed identically to Linear.
- Verdict: TUNE — investigate why selection count is always 0. If the virtual hit computation is a clone of another gun's trajectory, the gun is effectively dead weight. Confirm it fires unique virtual bullets.
Circular
What it does: Circular lead — assumes constant angular velocity (orbit).
- Strengths: RandomMover (80%), decent on most adversaries.
- Weaknesses: Lags behind Linear/WallBounce on Random; barely competitive elsewhere.
- Verdict: KEEP (marginal) — distinct from Linear (angular vs linear velocity assumption), worth keeping for orbit-heavy bots. Gets real selection only on RandomMover (350 ticks) and WaveSurfer (137).
GuessFactor
What it does: Guess-factor segmentation gun — divides lateral movement into bins and tracks historical hit density.
- Strengths: SittingDuck 98%, generally solid floor across all enemies.
- Weaknesses: Never edges above HeadOn on any adversary; OscillatorBot 63% (same floor). Needs history to warm up — may lag early rounds.
- Selection: Modest (100 RandomMover, 0 elsewhere). Learning gun that never became dominant.
- Verdict: KEEP — GF guns are the backbone of competitive targeting. Gets outperformed by AvgLead/DecayGF variants — investigate whether GF segmentation resolution needs tuning.
Pattern
What it does: Movement pattern matching — records enemy movement sequences and replays prediction.
- Strengths: SittingDuck 98%, PatternMover 95%, WaveSurfer 84% (when selected).
- Weaknesses: OscillatorBot 9% — catastrophically wrong; 1081 selection ticks wasted there. RandomMover 17% — random movement breaks pattern replay entirely.
- Critical bug: The selector gave Pattern 1081 ticks vs OscillatorBot despite a 9% rate. This means
MinObsBeforeCompetelet it compete before it had enough data to show a representative rate, or the rolling window allowed a lucky early patch to lock in selection. - Verdict: TUNE — the gun itself is sound against true pattern movers; the problem is premature selection. Raise
MinObsBeforeCompeteor add a "minimum rounds active" gate before Pattern can dominate the selector.
AntiSurf
What it does: Anti-surfing gun — attempts to predict dodge direction by modelling bullet-wave response.
- Strengths: None observed. 0% virtual hit rate against every adversary in every round.
- Weaknesses: Broken or misconfigured — 0% is not a statistical artifact; it is a systematic failure. Scored 147 selection ticks vs OscillatorBot and 19 vs PatternMover despite 0% rate.
- Verdict: DROP or fix before retest. 0% vhit means virtual bullets are being fired in the wrong direction entirely. The selection ticks it received represent rounds where it was preferred over functional guns — a liability.
WallBounce
What it does: Predicts enemy movement reflecting off walls.
- Strengths: SittingDuck 99% (joint best), RandomMover 82% (tied best), OscillatorBot 64%, PatternMover 95%, WaveSurfer 85%.
- Weaknesses: None significant — consistently top-tier across all adversary types.
- Selection: Gets real ticks on SittingDuck (68), OscillatorBot (82), RandomMover (215), PatternMover (211), WaveSurfer (51) — the selector does pick it when scores differentiate.
- Verdict: KEEP — one of the most consistent guns in the rack. Wall-aware prediction generalises well.
Accel
What it does: Acceleration-based lead — accounts for changing velocity.
- Strengths: SittingDuck 99%, PatternMover 96%, WaveSurfer 86% (joint best). Strong against momentum-based movers.
- Weaknesses: RandomMover 19% — severe drop. Random heading changes invalidate acceleration extrapolation completely.
- Selection: 281 ticks OscillatorBot, 255 WaveSurfer, 152 RandomMover. Gets selection time on Oscillator despite 63% (same as HeadOn) — selector noise.
- Verdict: KEEP but note the RandomMover cliff. The 19% rate while receiving 152 selection ticks vs Random is wasteful — the selector should have dropped it faster.
StopShot
What it does: Fires at the predicted stop point — targets deceleration patterns.
- Strengths: WaveSurfer 86% (joint best), PatternMover 96%, SittingDuck 99%, RandomMover 82%. Broad competence.
- Weaknesses: Gets almost zero selection despite top-tier rates (1 tick on most matchups). Under-selected almost everywhere.
- Verdict: KEEP — likely under-selected because it ties with AvgLead but has fewer early observations. Its consistent top performance suggests it should get more selection time.
Displace
What it does: Displacement-based prediction — fires at position + expected displacement vector.
- Strengths: PatternMover 96% (joint best), WaveSurfer 86%, SittingDuck 99%.
- Weaknesses: RandomMover 81% (slightly below Linear/WallBounce).
- Selection: Gets decent ticks on PatternMover (243) — the selector finds it there.
- Verdict: KEEP — solid and well-selected where it's strong.
AvgLead
What it does: Averaged lead angle over a window — smoothed velocity prediction.
- Strengths: OscillatorBot 65% (best gun), WaveSurfer 86% (joint best), PatternMover 96%, RandomMover 82%, SittingDuck 99%.
- Weaknesses: None — top or joint-top across all five adversaries.
- Selection: Gets consistent moderate ticks everywhere (97–110), indicating the selector finds it reliable.
- Verdict: KEEP — the most consistently high-performing gun in the rack. If only one gun were kept, this would be it.
DecayGF
What it does: Guess-factor with exponential decay — recent history weighted more heavily.
- Strengths: WaveSurfer 86% (joint best), PatternMover 96%, SittingDuck 99%.
- Weaknesses: OscillatorBot 63% (same as HeadOn floor). Needs time to build a useful GF profile.
- Selection: Gets 30 ticks on WaveSurfer, 20 on PatternMover — modest but real.
- Verdict: KEEP — the decay weighting should help against adapting movers; verify the GF profile is being updated correctly.
KNN
What it does: K-nearest neighbours targeting — finds historical situations most similar to current and fires the historically best angle.
- Strengths: PatternMover 96% (joint best), WaveSurfer 85%, SittingDuck 99%.
- Weaknesses: OscillatorBot 63% (floor), needs history to warm up.
- Selection: 50 PatternMover, 24 WaveSurfer — gets some selection where it performs.
- Verdict: KEEP — KNN is a sound approach that should improve as history accumulates across rounds. Cold-start performance is expected to be weak.
Recommendations
1. Guns to DROP
| Gun | Reason |
|---|---|
| AntiSurf | 0% virtual hit rate against all adversaries. Not statistically bad — systematically broken. Received selection ticks on two matchups despite never hitting anything. Fix or remove; as-is it is pure negative value. |
2. Guns to TUNE
| Gun | What to change |
|---|---|
| Pattern | Raise MinObsBeforeCompete significantly (suggest ≥ 50 obs). Add a round-count gate: Pattern should not be eligible to dominate the selector until it has fired virtual bullets for at least 2 full rounds. The 1081-tick / 9% disaster vs OscillatorBot is entirely a selector gate failure. |
| Tsetlin | Investigate zero selection count. Check that virtual bullets are computed with a unique angle (distinct from Linear). If it is accidentally sharing a firing angle with another gun, it will never differentiate. |
| Accel | Its 19% vs RandomMover while receiving 152 selection ticks is wasteful. The rolling window should surface this faster — check if the window length (100 ticks) is too long for Accel's decay signal to flush bad data quickly. |
| GuessFactor | Investigate why it never edges above HeadOn. GF is a proven algorithm — if it is not beating HeadOn after 10 rounds vs PatternMover or WaveSurfer, the bin resolution or lateral-velocity calculation may be wrong. |
3. Selector Improvements
- Pattern over-selection:
MinObsBeforeCompete = 15is too low for a memory gun. Pattern needs hundreds of ticks to build a meaningful database. Gate it at 100+ observations or require a minimum database size before competing. - HeadOn selection dominance: HeadOn gets 1010–1470 ticks per matchup while marginally better guns (AvgLead, WallBounce) get 50–250. The selector may be hitting HeadOn early when all guns are equal, then staying there due to recency bias. Consider a uniform-random tiebreak when rates are within 1–2%.
- Accel vs Random: 152 wasted ticks at 19% vhit. The 100-tick rolling window should expose this; check if the window resets between rounds or carries over (if it carries over from a good round, bad-round data is diluted).
4. Missing Coverage
- True surfer with timing randomness: WaveSurfer is the closest to real tournament surfing bots, but AvgLead/StopShot/Accel all hit 86%. Consider testing against a stronger surfer variant to stress-test wave-aware guns.
- Fast random with perpendicular escape: Accel's 19% on RandomMover suggests fast lateral movers are poorly handled. No gun clearly dominates RandomMover — the 82% ceiling is shared by four guns, none clearly specialised.
- Anti-mirror bots: No adversary tests symmetric evasion. If AntiSurf is ever fixed, it needs a proper test target.
Bugs Found During Testing
1. onBulletHitBot → onBulletHit (critical)
Location: ModularBot event handler registration.
Problem: The callback method was named onBulletHitBot but the Robocode API fires onBulletHit. Real hit tracking was silently dead — the callback never fired, so realHits in the gun stats relied on whatever fallback was in place.
Impact: Per-gun real hit rates could not be attributed correctly. The realHits field in the JSONL reflects aggregate post-fix behaviour but per-gun real-hit breakdown was unavailable for this gauntlet. Virtual hit rates remain valid (they are computed independently of the event callback).
Fix: Rename to onBulletHit. Already corrected; re-run the gauntlet to get clean real-hit attribution per gun.
2. AntiSurf 0% vhit (potential implementation bug)
Observed: AntiSurf returns 0 virtual hits against every adversary including SittingDuck. SittingDuck does not move — any gun aimed at the current position should achieve near-100%. Implication: AntiSurf is not firing virtual bullets at the target position at all, or its angle computation produces NaN/wrong quadrant. This is a code bug, not a tuning issue. Suggested check: Print the computed firing angle for AntiSurf virtual bullets and verify it is within ±π/2 of the actual bearing to target.
3. Pattern gun gets selected during initial warm-up
Observed: Pattern accumulates 1081 selection ticks vs OscillatorBot at 9% vhit. With MinObsBeforeCompete = 15, it became eligible before its pattern database was meaningful, won a random tiebreak at 100% during early ticks (before any miss was recorded), then the rolling window retained that optimistic early rate too long.
Fix: See Recommendations §3.