SirStone
013b9fe01e
Three corrections, all prompted by later measurements: 1. The virtual-vs-real rank correlation is NOT robustly negative. Six independent Spearman measurements now exist (-0.374, +0.335, +0.522, -0.371, -0.073, -0.037) and the sign flips on large samples, so it is near zero on average. The honest headline is that virtual hit rate is a POOR RANKER, not an inverted one. The report said 'not weak - it is inverted' in six places; it now says so in none. The practical conclusion (do not trust it for ranking) is unchanged; the mechanism claimed was wrong. 2. The offline==online acceptance is FIXED, not flaky. Root cause was that the replay spawned gun 13 (TMSelect) while the live rack has it disabled, and the shared VirtualTracker ring is ORDER-SENSITIVE, so gun 13's extra 4 bullets/tick permuted the per-tick resolution order for every other gun and shifted the learning guns' observations. After closing gun 13's ready gate offline the live and offline KNN traces are byte-identical (904/904 lines, empty diff). 5/5 consecutive runs now report 12/12 exact with the death boundary included. Recorded with the lesson: a flaky proof was hiding a real bug. Also records the general A/B confound - disabling a gun removes its 4 spawns/tick from the shared ring, perturbing resolution order for the rest. 3. Pruning was tested and does NOT help, so the verdict for Tsetlin and Displace changes from an implied drop to BELOW OVERALL - KEEP. 15 paired runs: baseline 6.18%, Tsetlin-off 5.76%, Tsetlin+Displace-off 5.46%; paired permutation p=0.57 and p=0.21; distributions completely overlap; a non-surfer control showed no separation. Being below average does not justify removal. Also records the tie-break randomness fix, and quotes run counts with every rate (6.95% over 13 runs vs 6.18% over 15 runs, same binary) rather than presenting a single figure as definitive.
4.2 KiB
Gun Rack Summary — TL;DR
Date: 2026-09-21 · Bot: ModularBot (13 active guns + 1 disabled TM gate)
Shipped config: metric path, selector relative.
Ground truth: server-side real hit rate (events sidecar), per-run, with an
explicit overlap test. Scores are not used — they swing by a couple of hundred
points per run.
Boss: the unmodified DrussGT jar through tools/robocode_shim/.
Headline (13 runs vs DrussGT, 3,612 server-side shots): 6.95% real hit rate,
251 dmg/run. The same binary on a 15-run set measures 6.18% (events
6.16%), so always read a rate with its run count.
The one thing to know: virtual hit rate is a poor ranker, not a reliable
proxy for real hit rate. Its Spearman correlation with real hit rate is weak
and sign-unstable across run sets — −0.374 over 13 runs, but +0.335
over a 15-run paired baseline (same 13 guns, different but equally defensible
aggregation) and +0.522 on the 5-run relative+path set — i.e. near zero on
average, not reliably anti-correlated. An earlier version of this report
overstated it as an inversion; the later 15-run measurement refuted that. 16
candidate ranking rules all failed to beat the shipped config; what works is the
selector's floor/tie hedging (removing the floor: 5.08% / 175 dmg vs 6.95% /
251 dmg). Detail: gun_rack_analysis.md.
Verdict table
| Verdict | Gun | Real % | Virtual % |
|---|---|---|---|
| KEEP | Linear | 10.7 | 10.2 |
| KEEP | Circular | 9.9 | 11.9 |
| KEEP | KNN | 9.0 | 7.5 |
| KEEP | Pattern | 8.6 | 12.0 |
| KEEP | Accel | 7.3 | 12.1 |
| KEEP | AvgLead | 7.0 | 12.3 |
| MARGINAL | GuessFactor | 6.9 | 10.3 |
| MARGINAL | DecayGF | 6.4 | 9.2 |
| MARGINAL | WallBounce | 6.2 | 12.9 |
| MARGINAL | StopShot | 6.1 | 12.6 |
| KEEP (below overall; pruning no help) | Tsetlin | 5.8 | 12.9 |
| KEEP (below overall; pruning no help) | Displace | 5.3 | 12.3 |
| FLOOR — KEEP | HeadOn | 5.2 | 8.6 |
| DISABLED | TMSelect | — | — |
HeadOn is lowest but must stay: it is the floor fallback, and disabling the floor measurably hurt (5.08% / 175 dmg).
Keep the full rack. Tsetlin and Displace are below overall, but removing them was tested: 15 paired runs per variant gave 6.18% → 5.76% (Tsetlin disabled) and 5.46% (Tsetlin+Displace disabled), with fully overlapping per-run distributions and paired permutation p = 0.57 / 0.21. Being below overall does not justify removal.
Top actions
- Stop trusting the virtual metric as a ranker. It is a poor ranker — weak, sign-unstable across run sets (−0.374 over 13 runs vs +0.335 over 15), and near zero on average, not reliably anti-correlated. The selector survives on its floor/tie hedge, not on its ordering.
- Acceptance test is fixed, not flaky. The old 11/12 was a real replay bug (the replay spawned disabled gun 13, and the shared ring is order-sensitive, so it permuted every other gun's resolution order). It now mirrors the live rack: 5/5 runs byte-identical 12/12 with the death boundary included.
- Get a second adversary. Every per-gun verdict rests on one wave surfer; the KEEP/BELOW boundaries are matchup-specific and per-gun N is small (47–898 shots).
- Keep Tsetlin and Displace (pruning tested). Both are below overall on small N, but disabling Tsetlin (15 paired runs: 6.18% → 5.76%) and Tsetlin+Displace (→ 5.46%) was neutral-to-slightly-negative; keep the full rack. Tsetlin now learns (clauses 714→13.8 literals) but is not competitive — the regression head is a follow-up, not grounds for removal.
- Do not re-enable TMSelect until the gate-margin/label problem is fixed; it cost 7.47% → 5.59% despite showing real energy structure in its clauses.
- Real-hit-rate-driven selection is not viable yet — unselected guns get near-zero shots, so it needs forced exploration + shrinkage + thousands of shots per gun.
- Selector tie-break is now seeded. It had been effectively deterministic
(
randomize()reached only incidentally via the Tsetlin constructor); now an explicit startup seed plus aGUN_SELECTOR_SEEDoverride makes seeded runs reproducible and unseeded runs vary.