SirStone/SirRoboGarage
1
0
Fork 0
Code Issues 28 Pull Requests Actions Projects Releases Wiki Activity

SETTLED: Pattern alone is the best single gun IN GENERAL, not just vs DrussGT

Browse Source 
Closes the one-adversary caveat that blocked shipping `onlyPattern`. Two arms
(`full` vs `onlyPattern`, rack knobs only), one frozen binary from clean HEAD
(a54ae6a, sha256 04d63cd8...), 10 adversaries, 120 battles, real server-side hit
rate, exact two-sided permutation test per adversary.

  adversary      full %   onlyPattern %   diff    exact p   winner
  drussgt         6.88        9.99        +3.11   0.0023    Pattern
  corners        85.10       88.48        +3.38   0.0012    Pattern
  crazy          39.16       48.42        +9.26   0.0006    Pattern
  patternmover   59.97       67.36        +7.39   0.0159    Pattern
  spinbot        83.37       81.20        -2.17   0.4720    full (n.s.)
  ramfire        97.06       95.58        -1.48   0.2214    full (n.s.)
  randommover    46.06       43.15        -2.91   0.3968    full (n.s.)
  sittingduck    96.37       96.42        +0.04   0.9048    tie
  oscillator     65.13       67.08        +1.94   0.5238    tie
  wavesurfer     41.09       42.68        +1.59   0.5159    tie

Pattern significantly WINS on 4 adversaries, ties on 3, and the full rack's three
nominal "wins" are all NON-SIGNIFICANT and only on saturated bots (83-97% hit
rate, where any gun works). **The full rack never significantly beats Pattern
alone on any adversary.** DrussGT replicates across a different frozen binary
(full 6.88 vs 6.93 prior; onlyPattern 9.99 vs 10.36/10.78 prior).

HONEST LIMITATION on the opponents: the real classic SpinBot/Corners/Crazy/RamFire
JARS are NOT runnable through the shim - `tools/robocode_shim/BotHost.java`
hardcodes `loadClass("jk.mega.DrussGT")`, and generalising it needs source edits.
So those four are the Tank Royale SAMPLE-BOT PORTS (the same opponents the shim
README validated against classic captures), not the original jars. Only DrussGT is
a real classic jar via the shim. Stated explicitly rather than implied.

Also noted: `full` in the harness passes TR_RACK_<GUN>=off for every gun, which
yields an empty admitted set and hits the documented FULL fallback ("an empty
membership admits every gun") - i.e. the shipped all-`both` rack. Verified against
the source, not assumed. Liveness proven per arm ([rack] active=..., gun_stats
100% Pattern for onlyPattern) and owner identification validated in 120/120 battles.

Follow-up if a stricter generalisation is wanted: add a generic classic-bot host to
the shim (a source change) and re-run those four as real jars.
This commit is contained in:
Davide Cappellini
2026-09-22 01:49:57 +02:00
parent a54ae6a162
commit 394b3deeed
1 changed files with 134 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/selector_negative_value.md
+134 -3
View File
@@ -2,6 +2,11 @@
**Measured. The best single gun beats the full rack, and not by a little.** **Measured. The best single gun beats the full rack, and not by a little.**
> **GENERALISED (this update):** the "vs DrussGT only" caveat is closed. Against **ten** adversaries,
> `Pattern` alone **significantly beats** the full rack on DrussGT, Corners, Crazy and PatternMover,
> **ties** on the other six, and **never significantly loses**. Ship `onlyPattern` — see the
> [Generalisation](#generalisation-settled-pattern-alone-wins-or-ties-on-all-ten-adversaries) section.
| arm | runs | shots | hits | real % | dmg/run | per-run range | exact two-sided p vs `full` | | arm | runs | shots | hits | real % | dmg/run | per-run range | exact two-sided p vs `full` |
|---|---:|---:|---:|---:|---:|---|---:| |---|---:|---:|---:|---:|---:|---|---:|
| `full` (shipped rack) | 7 | 3898 | 270 | **6.93** | 159 | 2.54–10.37 | — | | `full` (shipped rack) | 7 | 3898 | 270 | **6.93** | 159 | 2.54–10.37 | — |
@@ -16,6 +21,130 @@ It also fires MORE shots (4582 vs 3898), so it dominates on rate *and* volume.
This is **not** "any single gun wins" — `full` beats `onlyLinear`, `onlyGF` and `onlyKNN`. This is **not** "any single gun wins" — `full` beats `onlyLinear`, `onlyGF` and `onlyKNN`.
It is specifically **"Pattern alone beats the rack"**. It is specifically **"Pattern alone beats the rack"**.
---
## Generalisation (SETTLED): `Pattern` alone wins or ties on ALL ten adversaries
Every number in the section above is against DrussGT. This section closes that caveat. Same protocol
(one frozen binary, rack knobs only, server-side events sidecar, exact two-sided permutation test on
per-run rates), two arms — `full` (shipped rack, control) and `onlyPattern` — against **ten** adversaries.
**MEASURED verdict.** `onlyPattern` **significantly beats** the full rack on **DrussGT** (+3.11 pp,
p=0.0023), **Corners** (+3.38 pp, p=0.0012), **Crazy** (+9.26 pp, p=0.0006) and **PatternMover**
(+7.39 pp, p=0.0159), and is **statistically indistinguishable** on the other six (p ≥ 0.22). It
**never significantly loses**. The full rack **never significantly beats** `Pattern` alone on any
adversary. The full rack's only nominal edges are on **saturated** bots where nearly every shot already
hits (SpinBot −2.17 pp p=0.47, RamFire −1.48 pp p=0.22, RandomMover −2.91 pp p=0.40) — none significant.
**Direct answer: `Pattern` is the best single gun in general, not only against DrussGT.** The selector
is not merely redundant vs this one boss — on the discriminating opponents (DrussGT 7→10 %, Crazy
39→48 %, Corners 85→88 %) removing it is a real, significant gain. There is no adversary in this set
that justifies keeping the selector.
**MEASURED:** every hit rate, damage figure, run count and exact p-value above; the `[rack] active=`
lines; the `Pattern`-100 % selection mix; the per-battle owner identification. **INFERRED:** that the
result generalises beyond this finite ten-bot sample to "best gun in general"; that the ~83–97 %
hit-rate bots (SpinBot, RamFire, SittingDuck) are *saturated* and therefore weak discriminators rather
than genuine ties; and that the Tank Royale sample ports are a fair stand-in for the classic bots they
are ported from.
### Adversaries actually available (what is a *real classic* opponent, and what is not)
- **`drussgt`** — the real classic jar through the `robocode_shim` bridge. This is the **only** classic
jar the shim can host today: `tools/robocode_shim/src/robocode_shim/BotHost.java` hardcodes
`loader.loadClass("jk.mega.DrussGT")`. Driving the real classic SpinBot/Corners/Crazy/RamFire jars
through the shim would require shim source edits, which this measurement job forbids — so they are
**not available as shim opponents**.
- **`spinbot`, `corners`, `crazy`, `ramfire`** — the Tank Royale **sample-bot ports**
(`/home/davide/Projects/tank-royale/sample-bots/java/build/archive/…`), i.e. the exact opponents §5.8
of `tools/robocode_shim/README.md` already validated against the classic captures. They are **ports,
not the original classic jars** — stated plainly rather than passed off as the real thing.
- **`sittingduck`, `oscillator`, `randommover`, `patternmover`, `wavesurfer`** — the in-repo adversaries
(`common_libs/test_framework/adversaries/`). Deliberately **weak**; included as cheap extra data
points, but they are not the evidence base.
### Result table
7 runs × 7 rounds for DrussGT and the four sample ports; 5 runs × 7 rounds for the five weak in-repo
bots. Real hit rate = server-side events sidecar (`fire`/`hit`), not the bot's own attribution.
| adversary | adversary class | arm | runs | shots | hits | real % | dmg/run | per-run range | exact p vs `full` |
|---|---|---:|---:|---:|---:|---:|---:|---|---:|
| `drussgt` | shim (real classic jar) | `full` | 7 | 4087 | 281 | 6.88 | 168 | 5.11-8.83 | — |
| `drussgt` | shim (real classic jar) | **`onlyPattern`** | 7 | 4285 | 428 | **9.99** | **253** | 7.84-11.42 | **0.0023** |
| `spinbot` | TR sample port | `full` | 7 | 848 | 707 | 83.37 | 656 | 74.66-90.23 | — |
| `spinbot` | TR sample port | **`onlyPattern`** | 7 | 883 | 717 | **81.20** | **666** | 77.37-90.48 | **0.4720** |
| `corners` | TR sample port | `full` | 7 | 1134 | 965 | 85.10 | 662 | 81.68-87.12 | — |
| `corners` | TR sample port | **`onlyPattern`** | 7 | 1146 | 1014 | **88.48** | **662** | 86.71-90.07 | **0.0012** |
| `crazy` | TR sample port | `full` | 7 | 1213 | 475 | 39.16 | 616 | 33.79-43.97 | — |
| `crazy` | TR sample port | **`onlyPattern`** | 7 | 1105 | 535 | **48.42** | **644** | 46.15-51.27 | **0.0006** |
| `ramfire` | TR sample port | `full` | 7 | 442 | 429 | 97.06 | 637 | 93.94-100.00 | — |
| `ramfire` | TR sample port | **`onlyPattern`** | 7 | 430 | 411 | **95.58** | **627** | 92.98-98.44 | **0.2214** |
| `sittingduck` | in-repo (weak) | `full` | 5 | 662 | 638 | 96.37 | 676 | 93.60-98.06 | — |
| `sittingduck` | in-repo (weak) | **`onlyPattern`** | 5 | 726 | 700 | **96.42** | **686** | 95.59-98.26 | **0.9048** |
| `oscillator` | in-repo (weak) | `full` | 5 | 674 | 439 | 65.13 | 678 | 61.08-69.30 | — |
| `oscillator` | in-repo (weak) | **`onlyPattern`** | 5 | 814 | 546 | **67.08** | **644** | 53.24-80.92 | **0.5238** |
| `randommover` | in-repo (weak) | `full` | 5 | 1016 | 468 | 46.06 | 654 | 38.35-51.85 | — |
| `randommover` | in-repo (weak) | **`onlyPattern`** | 5 | 1008 | 435 | **43.15** | **643** | 39.37-50.98 | **0.3968** |
| `patternmover` | in-repo (weak) | `full` | 5 | 697 | 418 | 59.97 | 647 | 56.25-63.71 | — |
| `patternmover` | in-repo (weak) | **`onlyPattern`** | 5 | 622 | 419 | **67.36** | **659** | 62.20-77.39 | **0.0159** |
| `wavesurfer` | in-repo (weak) | `full` | 5 | 859 | 353 | 41.09 | 576 | 36.14-48.23 | — |
| `wavesurfer` | in-repo (weak) | **`onlyPattern`** | 5 | 888 | 379 | **42.68** | **616** | 37.21-49.24 | **0.5159** |
### Per-run real hit rates (%)
```
drussgt full r1=8.37 r2=5.70 r3=7.21 r4=5.11 r5=6.08 r6=6.38 r7=8.83
drussgt onlyPattern r1=10.79 r2=10.79 r3=7.84 r4=8.90 r5=11.11 r6=9.51 r7=11.42
spinbot full r1=88.32 r2=81.73 r3=90.23 r4=74.66 r5=88.31 r6=82.03 r7=80.49
spinbot onlyPattern r1=90.48 r2=85.29 r3=78.17 r4=79.39 r5=80.77 r6=80.47 r7=77.37
corners full r1=86.01 r2=84.09 r3=87.12 r4=86.42 r5=86.33 r6=85.00 r7=81.68
corners onlyPattern r1=89.09 r2=88.32 r3=90.07 r4=88.16 r5=89.50 r6=87.70 r7=86.71
crazy full r1=38.46 r2=43.97 r3=41.42 r4=39.39 r5=39.20 r6=40.26 r7=33.79
crazy onlyPattern r1=46.15 r2=51.27 r3=46.30 r4=50.96 r5=46.90 r6=47.37 r7=49.70
ramfire full r1=98.28 r2=93.94 r3=95.24 r4=100.00 r5=98.48 r6=94.12 r7=100.00
ramfire onlyPattern r1=93.75 r2=98.18 r3=95.38 r4=95.31 r5=95.08 r6=98.44 r7=92.98
sittingduck full r1=97.37 r2=98.06 r3=97.14 r4=96.92 r5=93.60
sittingduck onlyPattern r1=96.64 r2=96.27 r3=95.76 r4=98.26 r5=95.59
oscillator full r1=68.97 r2=61.08 r3=65.00 r4=63.25 r5=69.30
oscillator onlyPattern r1=64.33 r2=53.24 r3=80.92 r4=73.08 r5=72.14
randommover full r1=51.85 r2=48.00 r3=38.35 r4=49.17 r5=47.34
randommover onlyPattern r1=46.94 r2=42.20 r3=50.98 r4=39.37 r5=40.68
patternmover full r1=60.61 r2=56.25 r3=58.50 r4=63.71 r5=61.33
patternmover onlyPattern r1=77.39 r2=67.96 r3=64.34 r4=62.20 r5=66.42
wavesurfer full r1=36.14 r2=40.24 r3=41.76 r4=41.21 r5=48.23
wavesurfer onlyPattern r1=49.24 r2=42.86 r3=40.38 r4=37.21 r5=47.09
```
### Liveness (each arm was live)
The `[rack]` line printed at process start confirms membership for every run:
```
full [rack] mode=1v1 active=FULL [rack] mode=melee active=FULL
onlyPattern [rack] mode=1v1 active=PATTERN [rack] mode=melee active=PATTERN
```
(`full` emits `TR_RACK_<GUN>=off` for every gun; the empty admitted set triggers the documented
`rackActive == "FULL"` fallback, which is behaviourally the shipped all-`both` rack — verified against
`selectShotPolicy`: "An empty membership admits every gun".) The selected-gun mix in `gun_stats.jsonl`
confirms `onlyPattern` selects `Pattern` **100 %** of ticks on every adversary, while `full` distributes
across the rack (e.g. vs DrussGT: HeadOn 25.5 %, Pattern 18.4 %, KNN 10.0 %, Accel 8.7 %, …).
### Protocol (reproduction)
- **One frozen binary for every arm and every adversary**, built from **clean `a54ae6a`** via
`git archive` (the working tree was dirty from other agents) → `/tmp/ModularBot_generalise`,
sha256 `04d63cd87ea834fc6f9c55efb037c511fb79883389259809d57228f6d7e25de1`, bot dir name `ModularBot`.
- Rack knobs only (`TR_RACK_<GUN>=off`); **no source edits**. 8 concurrent battles, distinct output
files. ModularBot owner id is **not** stable across runs (it flipped 1↔2 randomly), so the analysis
identifies it independently per battle (power-bin subset for the shim; subject `bulletsFired` count
otherwise). All 120 battles passed the identification check.
- DrussGT rows replicate the earlier runs across a different frozen binary (`full` 6.88 % vs the earlier
6.93 %; `onlyPattern` 9.99 % vs 10.36–10.78 %) — **MEASURED** cross-build stability.
- Harness adapted from `tools/ab/which_gun_*.sh` + `which_gun_analyze.py` (exact permutation test
reused verbatim); generalised copies at `/tmp/whichgun_gen/`.
## Follow-up: a small rack of GOOD guns STILL loses to `Pattern` alone ## Follow-up: a small rack of GOOD guns STILL loses to `Pattern` alone
The obvious objection to the result above — "the rack is bloated, so of course it loses; prune it" — was The obvious objection to the result above — "the rack is bloated, so of course it loses; prune it" — was
@@ -99,9 +228,11 @@ From the `full` arm's own selection mix and per-gun real rates:
## What this does NOT yet settle ## What this does NOT yet settle
- **One adversary.** Everything here is vs DrussGT. `Pattern` should be re-checked against other - **One adversary — SETTLED (generalisation section above): NO LONGER A CAVEAT.** The ten-adversary run
bots before it becomes the default on the strength of this alone. (Supporting evidence: an offline shows `onlyPattern` significantly beats the full rack on DrussGT, Corners, Crazy and PatternMover, ties
audit found `Pattern` is the only gun competitive in *every* distance/speed bucket.) on the other six, and never significantly loses. `Pattern` is the best single gun in general, not just
vs DrussGT. (Supporting evidence: an offline audit found `Pattern` is the only gun competitive in
*every* distance/speed bucket.)
- **Whether a SMALL good rack beats `Pattern` alone — SETTLED (follow-up above): NO.** A 6-arm - **Whether a SMALL good rack beats `Pattern` alone — SETTLED (follow-up above): NO.** A 6-arm
follow-up (`lean8`, `lean6`, `pairPC`, `pairPK`, `pairPL`) found every selector-active rack worse or tied; follow-up (`lean8`, `lean6`, `pairPC`, `pairPK`, `pairPL`) found every selector-active rack worse or tied;
`lean6` 8.83% and `lean8` 6.31% both lose to `Pattern` alone 10.36% (p = 0.026 / 0.017). The selector is `lean6` 8.83% and `lean8` 6.31% both lose to `Pattern` alone 10.36% (p = 0.026 / 0.017). The selector is