SirStone/SirRoboGarage
1
0
Fork 0
Code Issues 28 Pull Requests Actions Projects Releases Wiki Activity

gun j123 Batch 1 outcome + Batch 2 prereg: len6 is the only arm to beat pattern on wins (15 opp), but all 5 arms are wins-positive incl. the bearing-invariant rad_offset; pre-register the 33-opponent confirmation with a structural control

Browse Source
This commit is contained in:
Davide Cappellini
2026-09-26 04:22:49 +02:00
parent 6c21dc4c58
commit 1b59b65810
2 changed files with 179 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/gun_campaign.md
+145
View File
@@ -570,3 +570,148 @@ campaign rule 2 (one primary up at cross-opponent sign-test p<0.05 while the
other does not go down) **AND** its damage CI excludes 0 **AND** the sign-flip
permutation test gives p<0.05. Otherwise Batch 1 is the answer.
### Outcome — Batch 1
**MEASURED.** Session `/tmp/ab/j123_b1`, commit
`2a98aba91b0ec023f580cd2485cafba2404d275d`, frozen binary sha256 `287d292fb7d1…`,
**15 opponents × 6 arms × 3 runs × 3 rounds = 270 battles, 0 failed, 0 never
started, 0 liveness exclusions**. Arena serialization held: the parallel job
j124 (`/tmp/ab/j124_spinner.run.log`) detected this session's processes at
04:08–04:18 and **waited**; no foreign battle ran concurrently. Every arm
declared `TR_MOVEMENT=strafe`; the new `TR_PATTERN_LEN` / `TR_PATTERN_DEPTH`
tokens appear verbatim in each overriding arm's own raw-env report and the
reference arm shows the shipped defaults 10 / 500.
#### Pooled dashboard (all valid runs — explanation only, NOT the verdict)
| arm | runs | dmg/run | dmg taken/run | wins/run | round wins | win rate | incoming hit rate | mean distance |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| `pattern` (REF) | 45 | 101.1 | 161.0 | 1.27 | 57/135 | 42.2% | 13.27% | 428 |
| `len6` | 45 | 110.8 | 151.2 | **1.76** | **79/135** | **58.5%** | **12.28%** | 424 |
| `len16` | 45 | 103.3 | 158.8 | 1.53 | 69/135 | 51.1% | 13.07% | 427 |
| `depth100` | 45 | 99.9 | 154.5 | 1.53 | 69/135 | 51.1% | 12.77% | 429 |
| `rad_offset` | 45 | 107.9 | 157.2 | 1.69 | 76/135 | 56.3% | 13.21% | 431 |
| `rad_scale` | 45 | 105.9 | 157.1 | 1.38 | 62/135 | 45.9% | 12.62% | 429 |
#### Per-opponent paired deltas (arm − `pattern`), damage and wins
| opponent | style | Δdmg_len6 | Δwins_len6 | Δdmg_len16 | Δwins_len16 | Δdmg_depth100 | Δwins_depth100 | Δdmg_rad_offset | Δwins_rad_offset | Δdmg_rad_scale | Δwins_rad_scale |
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| DrussGT | dodger | +8.4 | +0.67 | -21.7 | -0.33 | +3.6 | +0.67 | +10.5 | +1.33 | +18.7 | +0.67 |
| Diamond | dodger | -22.3 | +0.00 | -18.1 | -0.33 | -54.7 | -0.33 | -30.9 | +0.00 | -17.0 | -0.33 |
| Dookious | dodger | -6.6 | +1.33 | +7.6 | +1.00 | -0.8 | +0.67 | +2.6 | +0.67 | -15.6 | -0.33 |
| GresSuffurd | dodger | +32.3 | +0.00 | +31.9 | +0.00 | -6.6 | +0.00 | +28.4 | +0.67 | -0.9 | +0.00 |
| CassiusClay | dodger | +15.0 | +0.67 | +6.8 | +1.00 | +10.0 | +0.33 | +12.5 | +0.67 | +40.4 | +1.00 |
| RetroGirl | pattern | +48.0 | +1.33 | +20.3 | +0.67 | +59.9 | +1.00 | +4.1 | +0.00 | +55.5 | +0.33 |
| TripHammer | pattern | +6.1 | +0.00 | +0.1 | +0.33 | -21.6 | +0.00 | -9.3 | +0.00 | +4.7 | +0.33 |
| Coriantumr | pattern | +1.9 | -0.33 | -3.9 | +0.33 | +2.0 | +1.00 | -5.3 | -0.33 | -10.3 | -0.33 |
| WallAvoider | wallfollower | +28.5 | +1.00 | -23.5 | +0.00 | +41.0 | +0.00 | +23.6 | +1.00 | -24.0 | +0.00 |
| HawkOnFire | cornercamper | -8.9 | +0.67 | -9.5 | +0.00 | -9.3 | -0.33 | -12.0 | +0.33 | -5.0 | +0.33 |
| SpinBot | spinner | +5.7 | +0.00 | +13.3 | +0.00 | -13.5 | +0.00 | -1.7 | +0.00 | -0.3 | +0.00 |
| DiamondStealer | rammer | -16.1 | +0.00 | -9.8 | +0.00 | -14.6 | +0.00 | +31.8 | +1.33 | +1.1 | -0.67 |
| BlitzBat | brawler | +9.1 | +0.00 | +17.8 | +0.33 | -6.3 | +0.33 | +9.6 | +0.33 | +9.2 | -0.67 |
| YersiniaPestis | aggressive | +21.0 | +1.33 | +2.6 | +0.67 | -18.6 | +0.33 | +27.4 | +0.67 | +3.0 | +0.67 |
| Ascendant | aggressive | +22.4 | +0.67 | +18.7 | +0.33 | +10.9 | +0.33 | +9.3 | -0.33 | +12.6 | +0.67 |
#### Cross-opponent aggregation (the verdict layer, damage + wins)
| arm | metric | mean Δ | spread (SD) | SE | 95% CI | sign test (wins/n) | p(sign) | p(sign-flip) | Wilcoxon p | MDE |
|---|---|---:|---:|---:|---|---:|---:|---:|---:|---:|
| `len6` | damage | +9.63 | 18.97 | 4.90 | [−0.88, +20.13] | 11/15 | 0.1185 | 0.06976 | 0.09384 | 13.72 |
| `len6` | wins | **+0.49** | 0.58 | 0.15 | **[+0.17, +0.81]** | 8/9 | **0.03906** | **0.007812** | **0.01269** | 0.42 |
| `len16` | damage | +2.18 | 16.62 | 4.29 | [−7.02, +11.38] | 9/15 | 0.6072 | 0.6142 | 0.712 | 12.02 |
| `len16` | wins | +0.27 | 0.42 | 0.11 | [+0.03, +0.50] | 8/10 | 0.1094 | 0.04688 | 0.01852 | 0.30 |
| `depth100` | damage | −1.24 | 26.52 | 6.85 | [−15.93, +13.45] | 6/15 | 0.6072 | 0.8677 | 0.5137 | 19.19 |
| `depth100` | wins | +0.27 | 0.42 | 0.11 | [+0.03, +0.50] | 8/10 | 0.1094 | 0.04688 | 0.046 | 0.30 |
| `rad_offset` | damage | +6.71 | 17.18 | 4.44 | [−2.80, +16.23] | 10/15 | 0.3018 | 0.1522 | 0.1323 | 12.43 |
| `rad_offset` | wins | +0.42 | 0.54 | 0.14 | [+0.12, +0.72] | 9/11 | 0.06543 | 0.01465 | 0.01424 | 0.39 |
| `rad_scale` | damage | +4.80 | 21.10 | 5.45 | [−6.89, +16.49] | 8/15 | 1 | 0.4095 | 0.6293 | 15.27 |
| `rad_scale` | wins | +0.11 | 0.51 | 0.13 | [−0.17, +0.40] | 7/12 | 0.7744 | 0.5107 | 0.289 | 0.37 |
#### The pre-registered verdict (verbatim)
| rank | arm | Δwins/run | Δdmg/run | sign test wins | sign test dmg | verdict (strict) | verdict (substantive) |
|---:|---|---:|---:|---|---|---|---|
| 1 | `len6` | +0.49 | +9.6 | 8/9 p=0.03906 | 11/15 p=0.1185 | **BETTER** | **BETTER** |
| 2 | `rad_offset` | +0.42 | +6.7 | 9/11 p=0.06543 | 10/15 p=0.3018 | **not distinguishable** | **not distinguishable** |
| 3 | `len16` | +0.27 | +2.2 | 8/10 p=0.1094 | 9/15 p=0.6072 | **not distinguishable** | **not distinguishable** |
| 4 | `depth100` | +0.27 | −1.2 | 8/10 p=0.1094 | 6/15 p=0.6072 | **not distinguishable** | **not distinguishable** |
| 5 | `rad_scale` | +0.11 | +4.8 | 7/12 p=0.7744 | 8/15 p=1 | **not distinguishable** | **not distinguishable** |
Reference `pattern`: 101.1 dmg/run, 1.27 wins/run, 13.27% incoming, 428 px.
#### Reading and the single red flag
* **`len6` is the only arm that beats `pattern` by the campaign rule** — on the
WINS primary: +0.49 wins/run, sign test 8/9 p=0.0391, sign-flip p=0.0078,
wins CI [+0.17, +0.81] excluding 0 — while damage is nominally up (+9.6, but
inside the 13.7 MDE and its CI includes 0). A 6-tick match key is broadly
positive (10/15 opponents non-negative, only `Coriantumr` −0.33 negative).
* **RED FLAG (MEASURED): all five arms are nominally positive on wins**
(+0.49 / +0.27 / +0.27 / +0.42 / +0.11), including `rad_offset`, which by job
j99's exact structural proof **cannot change the aim bearing** and whose live
effect was measured as a tie (+0.28 pp, p=0.62). The reference `pattern` also
measures **42.2% wins here vs 50.4% in j121 on the same 15-panel** (its damage
is unchanged at 101–99.6), so the leading explanation for the all-arms-positive
pattern is a **session-level low reference**, not five independent gun wins.
The within-session paired deltas remain valid, but `len6`'s win is exactly the
kind of single-session effect that must replicate on a second, wider panel.
* **The two shape knobs behave plausibly.** `len16` (fewer, more specific
matches → more linear fallback) is damage-flat; `depth100` (shallower search)
is damage-flat at a larger MDE. Neither is a detectable win.
#### Pre-registered predictions — scorecard
| # | prediction | outcome |
|---|---|---|
| 1 | no arm beats `pattern` on BOTH primaries | **CORRECT** (`len6` beats on wins only; damage CI includes 0) |
| 2 | `len16` flat/worse on damage, wins flat | **PARTLY WRONG**: damage flat (+2.2), wins nominally +0.27 |
| 3 | `len6` flat on damage and wins | **WRONG**: wins +0.49 significant; damage +9.6 sub-MDE |
| 4 | `depth100` not distinguishable | **CORRECT** (damage −1.2), though wins nominally +0.27 |
| 5 | radial controls not distinguishable / bearing-invariant | **CORRECT**: `rad_scale` flat; `rad_offset` nominal +0.42 wins but sign p=0.065 (not significant) |
| 6 | any surprise is damage-only without wins | **WRONG in direction**: the surprise is wins-with-nominal-damage |
## Batch 2 — higher-power confirmation of `len6` (+ the structural control)
**Why we run it (task trigger, with one recorded deviation).** The task's Batch-2
trigger is: an arm beats the reference with **the winning primary's CI excluding
0 AND sign-flip p<0.05**. `len6` satisfies it on wins (CI [+0.17, +0.81];
p(sign-flip)=0.0078). *Deviation, recorded:* the Batch-1 pre-registration added a
stricter clause — "its **damage** CI excludes 0" — which `len6`'s damage
([−0.88, +20.13]) does not meet. We run Batch 2 anyway because (a) the task rule
keys on the **winning** primary and (b) the all-arms-positive red flag means a
replication is the only way to tell a real `len6` effect from a session-level
reference anomaly. This is a deviation from the letter of my own pre-registration,
not from the task's criterion.
**Design.** One frozen binary, four env-only arms, the wider panel
`tools/ab/panel_gun_b2.txt` (**33 opponents**, superset of Batch 1's 15),
**3 runs × 3 rounds = 396 battles**, `--conc 6`, `--wait-arena`. Arm file:
`tools/ab/arms_gun_b4.txt`. Reference: `pattern`. Movement pinned
`TR_MOVEMENT=strafe` in every arm.
| # | arm | env over the pin | role |
|---|---|---|---|
| 1 | `pattern` | (none) | reference |
| 2 | `len6` | `TR_PATTERN_LEN=6` | the candidate |
| 3 | `len16` | `TR_PATTERN_LEN=16` | dose-response on key length |
| 4 | `rad_offset` | `TR_PATTERN_RAD_OFFSET=-20` | **structural positive control** (bearing-invariant) |
**Pre-registered prediction (written BEFORE the battles finished):**
1. **`len6`'s +0.49 wins does NOT replicate.** On 33 opponents its wins delta
collapses toward the MDE and the sign test (or the sign-flip test) fails.
Rationale: all five Batch-1 arms were positive on wins including the
bearing-invariant `rad_offset`, and the reference was 8 points low on win rate
versus j121's same-panel measurement — a single-session low reference, not a
`len6` effect.
2. **`rad_offset` is not a detectable win**; if it, too, replicates a wins gain,
the wins axis is being moved by something other than the aim bearing and the
whole Batch-1 wins signal is an artifact.
3. `len16` stays within its MDE on both primaries.
4. Damage stays within the MDE for every arm.
5. **If `len6` DOES replicate** (winning-metric CI excludes 0, sign-flip
p<0.05, damage not detectably down), it is recorded as the campaign's
**champion candidate** — and the shipped-default flip remains a separate,
explicit decision that is **NOT** taken in this job.
tools/ab/arms_gun_b4.txt
+34
View File
@@ -0,0 +1,34 @@
# ─────────────────────────────────────────────────────────────────────────────
# arms_gun_b4.txt — PHASE 2 BATCH 2 of the GUN campaign: higher-power
# confirmation of the Batch-1 `len6` wins signal, on the wider 33-opponent
# panel, with the bearing-invariant `rad_offset` as a structural control.
#
# Format: name | ENV=value ENV=value | label
#
# ONE frozen binary (tournament_run.sh builds it from `git archive HEAD`); every
# arm below differs ONLY by its env dict. No per-arm rebuild.
#
# MOVEMENT IS PINNED IN EVERY ARM (`TR_MOVEMENT=strafe`) — same rationale as
# Batch 1 (movement-default / concurrent-job hygiene + analyzer liveness rule).
#
# Why `rad_offset` is here: job j99 proved exactly that the radial correction
# changes only the aim DISTANCE and leaves the BEARING untouched, so it cannot
# move the real shot direction; its live effect was a tie (+0.28 pp, p=0.62). In
# Batch 1 it nonetheless showed +0.42 wins/run (sign test p=0.065) — and so did
# ALL five arms, including this structural no-op. If it repeats the nominal wins
# gain on 33 opponents, the Batch-1 wins signal is a session-level reference
# anomaly rather than a gun effect.
# ─────────────────────────────────────────────────────────────────────────────
# 1. REFERENCE. The shipped `onlyPattern` rack, movement pinned, no shape env.
pattern | TR_MOVEMENT=strafe | shipped onlyPattern rack (REFERENCE)
# 2. The Batch-1 candidate: 6-tick match key (+0.49 wins/run on 15 opponents).
len6 | TR_MOVEMENT=strafe TR_PATTERN_LEN=6 | match key = 6 ticks (Batch-1 candidate)
# 3. Dose-response on the key-length axis.
len16 | TR_MOVEMENT=strafe TR_PATTERN_LEN=16 | match key = 16 ticks
# 4. STRUCTURAL POSITIVE CONTROL: bearing-invariant radial knob; cannot change
# the real aim direction (job j99), so a hits/wins gain here is an artifact.
rad_offset | TR_MOVEMENT=strafe TR_PATTERN_RAD_OFFSET=-20 | structural control: aim short 20 px