diff --git a/docs/gun_campaign.md b/docs/gun_campaign.md index fcc62fa..1248bfc 100644 --- a/docs/gun_campaign.md +++ b/docs/gun_campaign.md @@ -570,3 +570,148 @@ campaign rule 2 (one primary up at cross-opponent sign-test p<0.05 while the other does not go down) **AND** its damage CI excludes 0 **AND** the sign-flip permutation test gives p<0.05. Otherwise Batch 1 is the answer. + +### Outcome — Batch 1 + +**MEASURED.** Session `/tmp/ab/j123_b1`, commit +`2a98aba91b0ec023f580cd2485cafba2404d275d`, frozen binary sha256 `287d292fb7d1…`, +**15 opponents × 6 arms × 3 runs × 3 rounds = 270 battles, 0 failed, 0 never +started, 0 liveness exclusions**. Arena serialization held: the parallel job +j124 (`/tmp/ab/j124_spinner.run.log`) detected this session's processes at +04:08–04:18 and **waited**; no foreign battle ran concurrently. Every arm +declared `TR_MOVEMENT=strafe`; the new `TR_PATTERN_LEN` / `TR_PATTERN_DEPTH` +tokens appear verbatim in each overriding arm's own raw-env report and the +reference arm shows the shipped defaults 10 / 500. + +#### Pooled dashboard (all valid runs — explanation only, NOT the verdict) + +| arm | runs | dmg/run | dmg taken/run | wins/run | round wins | win rate | incoming hit rate | mean distance | +|---|---:|---:|---:|---:|---:|---:|---:|---:| +| `pattern` (REF) | 45 | 101.1 | 161.0 | 1.27 | 57/135 | 42.2% | 13.27% | 428 | +| `len6` | 45 | 110.8 | 151.2 | **1.76** | **79/135** | **58.5%** | **12.28%** | 424 | +| `len16` | 45 | 103.3 | 158.8 | 1.53 | 69/135 | 51.1% | 13.07% | 427 | +| `depth100` | 45 | 99.9 | 154.5 | 1.53 | 69/135 | 51.1% | 12.77% | 429 | +| `rad_offset` | 45 | 107.9 | 157.2 | 1.69 | 76/135 | 56.3% | 13.21% | 431 | +| `rad_scale` | 45 | 105.9 | 157.1 | 1.38 | 62/135 | 45.9% | 12.62% | 429 | + +#### Per-opponent paired deltas (arm − `pattern`), damage and wins + +| opponent | style | Δdmg_len6 | Δwins_len6 | Δdmg_len16 | Δwins_len16 | Δdmg_depth100 | Δwins_depth100 | Δdmg_rad_offset | Δwins_rad_offset | Δdmg_rad_scale | Δwins_rad_scale | +|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| +| DrussGT | dodger | +8.4 | +0.67 | -21.7 | -0.33 | +3.6 | +0.67 | +10.5 | +1.33 | +18.7 | +0.67 | +| Diamond | dodger | -22.3 | +0.00 | -18.1 | -0.33 | -54.7 | -0.33 | -30.9 | +0.00 | -17.0 | -0.33 | +| Dookious | dodger | -6.6 | +1.33 | +7.6 | +1.00 | -0.8 | +0.67 | +2.6 | +0.67 | -15.6 | -0.33 | +| GresSuffurd | dodger | +32.3 | +0.00 | +31.9 | +0.00 | -6.6 | +0.00 | +28.4 | +0.67 | -0.9 | +0.00 | +| CassiusClay | dodger | +15.0 | +0.67 | +6.8 | +1.00 | +10.0 | +0.33 | +12.5 | +0.67 | +40.4 | +1.00 | +| RetroGirl | pattern | +48.0 | +1.33 | +20.3 | +0.67 | +59.9 | +1.00 | +4.1 | +0.00 | +55.5 | +0.33 | +| TripHammer | pattern | +6.1 | +0.00 | +0.1 | +0.33 | -21.6 | +0.00 | -9.3 | +0.00 | +4.7 | +0.33 | +| Coriantumr | pattern | +1.9 | -0.33 | -3.9 | +0.33 | +2.0 | +1.00 | -5.3 | -0.33 | -10.3 | -0.33 | +| WallAvoider | wallfollower | +28.5 | +1.00 | -23.5 | +0.00 | +41.0 | +0.00 | +23.6 | +1.00 | -24.0 | +0.00 | +| HawkOnFire | cornercamper | -8.9 | +0.67 | -9.5 | +0.00 | -9.3 | -0.33 | -12.0 | +0.33 | -5.0 | +0.33 | +| SpinBot | spinner | +5.7 | +0.00 | +13.3 | +0.00 | -13.5 | +0.00 | -1.7 | +0.00 | -0.3 | +0.00 | +| DiamondStealer | rammer | -16.1 | +0.00 | -9.8 | +0.00 | -14.6 | +0.00 | +31.8 | +1.33 | +1.1 | -0.67 | +| BlitzBat | brawler | +9.1 | +0.00 | +17.8 | +0.33 | -6.3 | +0.33 | +9.6 | +0.33 | +9.2 | -0.67 | +| YersiniaPestis | aggressive | +21.0 | +1.33 | +2.6 | +0.67 | -18.6 | +0.33 | +27.4 | +0.67 | +3.0 | +0.67 | +| Ascendant | aggressive | +22.4 | +0.67 | +18.7 | +0.33 | +10.9 | +0.33 | +9.3 | -0.33 | +12.6 | +0.67 | + +#### Cross-opponent aggregation (the verdict layer, damage + wins) + +| arm | metric | mean Δ | spread (SD) | SE | 95% CI | sign test (wins/n) | p(sign) | p(sign-flip) | Wilcoxon p | MDE | +|---|---|---:|---:|---:|---|---:|---:|---:|---:|---:| +| `len6` | damage | +9.63 | 18.97 | 4.90 | [−0.88, +20.13] | 11/15 | 0.1185 | 0.06976 | 0.09384 | 13.72 | +| `len6` | wins | **+0.49** | 0.58 | 0.15 | **[+0.17, +0.81]** | 8/9 | **0.03906** | **0.007812** | **0.01269** | 0.42 | +| `len16` | damage | +2.18 | 16.62 | 4.29 | [−7.02, +11.38] | 9/15 | 0.6072 | 0.6142 | 0.712 | 12.02 | +| `len16` | wins | +0.27 | 0.42 | 0.11 | [+0.03, +0.50] | 8/10 | 0.1094 | 0.04688 | 0.01852 | 0.30 | +| `depth100` | damage | −1.24 | 26.52 | 6.85 | [−15.93, +13.45] | 6/15 | 0.6072 | 0.8677 | 0.5137 | 19.19 | +| `depth100` | wins | +0.27 | 0.42 | 0.11 | [+0.03, +0.50] | 8/10 | 0.1094 | 0.04688 | 0.046 | 0.30 | +| `rad_offset` | damage | +6.71 | 17.18 | 4.44 | [−2.80, +16.23] | 10/15 | 0.3018 | 0.1522 | 0.1323 | 12.43 | +| `rad_offset` | wins | +0.42 | 0.54 | 0.14 | [+0.12, +0.72] | 9/11 | 0.06543 | 0.01465 | 0.01424 | 0.39 | +| `rad_scale` | damage | +4.80 | 21.10 | 5.45 | [−6.89, +16.49] | 8/15 | 1 | 0.4095 | 0.6293 | 15.27 | +| `rad_scale` | wins | +0.11 | 0.51 | 0.13 | [−0.17, +0.40] | 7/12 | 0.7744 | 0.5107 | 0.289 | 0.37 | + +#### The pre-registered verdict (verbatim) + +| rank | arm | Δwins/run | Δdmg/run | sign test wins | sign test dmg | verdict (strict) | verdict (substantive) | +|---:|---|---:|---:|---|---|---|---| +| 1 | `len6` | +0.49 | +9.6 | 8/9 p=0.03906 | 11/15 p=0.1185 | **BETTER** | **BETTER** | +| 2 | `rad_offset` | +0.42 | +6.7 | 9/11 p=0.06543 | 10/15 p=0.3018 | **not distinguishable** | **not distinguishable** | +| 3 | `len16` | +0.27 | +2.2 | 8/10 p=0.1094 | 9/15 p=0.6072 | **not distinguishable** | **not distinguishable** | +| 4 | `depth100` | +0.27 | −1.2 | 8/10 p=0.1094 | 6/15 p=0.6072 | **not distinguishable** | **not distinguishable** | +| 5 | `rad_scale` | +0.11 | +4.8 | 7/12 p=0.7744 | 8/15 p=1 | **not distinguishable** | **not distinguishable** | + +Reference `pattern`: 101.1 dmg/run, 1.27 wins/run, 13.27% incoming, 428 px. + +#### Reading and the single red flag + +* **`len6` is the only arm that beats `pattern` by the campaign rule** — on the + WINS primary: +0.49 wins/run, sign test 8/9 p=0.0391, sign-flip p=0.0078, + wins CI [+0.17, +0.81] excluding 0 — while damage is nominally up (+9.6, but + inside the 13.7 MDE and its CI includes 0). A 6-tick match key is broadly + positive (10/15 opponents non-negative, only `Coriantumr` −0.33 negative). +* **RED FLAG (MEASURED): all five arms are nominally positive on wins** + (+0.49 / +0.27 / +0.27 / +0.42 / +0.11), including `rad_offset`, which by job + j99's exact structural proof **cannot change the aim bearing** and whose live + effect was measured as a tie (+0.28 pp, p=0.62). The reference `pattern` also + measures **42.2% wins here vs 50.4% in j121 on the same 15-panel** (its damage + is unchanged at 101–99.6), so the leading explanation for the all-arms-positive + pattern is a **session-level low reference**, not five independent gun wins. + The within-session paired deltas remain valid, but `len6`'s win is exactly the + kind of single-session effect that must replicate on a second, wider panel. +* **The two shape knobs behave plausibly.** `len16` (fewer, more specific + matches → more linear fallback) is damage-flat; `depth100` (shallower search) + is damage-flat at a larger MDE. Neither is a detectable win. + +#### Pre-registered predictions — scorecard + +| # | prediction | outcome | +|---|---|---| +| 1 | no arm beats `pattern` on BOTH primaries | **CORRECT** (`len6` beats on wins only; damage CI includes 0) | +| 2 | `len16` flat/worse on damage, wins flat | **PARTLY WRONG**: damage flat (+2.2), wins nominally +0.27 | +| 3 | `len6` flat on damage and wins | **WRONG**: wins +0.49 significant; damage +9.6 sub-MDE | +| 4 | `depth100` not distinguishable | **CORRECT** (damage −1.2), though wins nominally +0.27 | +| 5 | radial controls not distinguishable / bearing-invariant | **CORRECT**: `rad_scale` flat; `rad_offset` nominal +0.42 wins but sign p=0.065 (not significant) | +| 6 | any surprise is damage-only without wins | **WRONG in direction**: the surprise is wins-with-nominal-damage | + +## Batch 2 — higher-power confirmation of `len6` (+ the structural control) + +**Why we run it (task trigger, with one recorded deviation).** The task's Batch-2 +trigger is: an arm beats the reference with **the winning primary's CI excluding +0 AND sign-flip p<0.05**. `len6` satisfies it on wins (CI [+0.17, +0.81]; +p(sign-flip)=0.0078). *Deviation, recorded:* the Batch-1 pre-registration added a +stricter clause — "its **damage** CI excludes 0" — which `len6`'s damage +([−0.88, +20.13]) does not meet. We run Batch 2 anyway because (a) the task rule +keys on the **winning** primary and (b) the all-arms-positive red flag means a +replication is the only way to tell a real `len6` effect from a session-level +reference anomaly. This is a deviation from the letter of my own pre-registration, +not from the task's criterion. + +**Design.** One frozen binary, four env-only arms, the wider panel +`tools/ab/panel_gun_b2.txt` (**33 opponents**, superset of Batch 1's 15), +**3 runs × 3 rounds = 396 battles**, `--conc 6`, `--wait-arena`. Arm file: +`tools/ab/arms_gun_b4.txt`. Reference: `pattern`. Movement pinned +`TR_MOVEMENT=strafe` in every arm. + +| # | arm | env over the pin | role | +|---|---|---|---| +| 1 | `pattern` | (none) | reference | +| 2 | `len6` | `TR_PATTERN_LEN=6` | the candidate | +| 3 | `len16` | `TR_PATTERN_LEN=16` | dose-response on key length | +| 4 | `rad_offset` | `TR_PATTERN_RAD_OFFSET=-20` | **structural positive control** (bearing-invariant) | + +**Pre-registered prediction (written BEFORE the battles finished):** +1. **`len6`'s +0.49 wins does NOT replicate.** On 33 opponents its wins delta + collapses toward the MDE and the sign test (or the sign-flip test) fails. + Rationale: all five Batch-1 arms were positive on wins including the + bearing-invariant `rad_offset`, and the reference was 8 points low on win rate + versus j121's same-panel measurement — a single-session low reference, not a + `len6` effect. +2. **`rad_offset` is not a detectable win**; if it, too, replicates a wins gain, + the wins axis is being moved by something other than the aim bearing and the + whole Batch-1 wins signal is an artifact. +3. `len16` stays within its MDE on both primaries. +4. Damage stays within the MDE for every arm. +5. **If `len6` DOES replicate** (winning-metric CI excludes 0, sign-flip + p<0.05, damage not detectably down), it is recorded as the campaign's + **champion candidate** — and the shipped-default flip remains a separate, + explicit decision that is **NOT** taken in this job. + diff --git a/tools/ab/arms_gun_b4.txt b/tools/ab/arms_gun_b4.txt new file mode 100644 index 0000000..7c17a27 --- /dev/null +++ b/tools/ab/arms_gun_b4.txt @@ -0,0 +1,34 @@ +# ───────────────────────────────────────────────────────────────────────────── +# arms_gun_b4.txt — PHASE 2 BATCH 2 of the GUN campaign: higher-power +# confirmation of the Batch-1 `len6` wins signal, on the wider 33-opponent +# panel, with the bearing-invariant `rad_offset` as a structural control. +# +# Format: name | ENV=value ENV=value | label +# +# ONE frozen binary (tournament_run.sh builds it from `git archive HEAD`); every +# arm below differs ONLY by its env dict. No per-arm rebuild. +# +# MOVEMENT IS PINNED IN EVERY ARM (`TR_MOVEMENT=strafe`) — same rationale as +# Batch 1 (movement-default / concurrent-job hygiene + analyzer liveness rule). +# +# Why `rad_offset` is here: job j99 proved exactly that the radial correction +# changes only the aim DISTANCE and leaves the BEARING untouched, so it cannot +# move the real shot direction; its live effect was a tie (+0.28 pp, p=0.62). In +# Batch 1 it nonetheless showed +0.42 wins/run (sign test p=0.065) — and so did +# ALL five arms, including this structural no-op. If it repeats the nominal wins +# gain on 33 opponents, the Batch-1 wins signal is a session-level reference +# anomaly rather than a gun effect. +# ───────────────────────────────────────────────────────────────────────────── + +# 1. REFERENCE. The shipped `onlyPattern` rack, movement pinned, no shape env. +pattern | TR_MOVEMENT=strafe | shipped onlyPattern rack (REFERENCE) + +# 2. The Batch-1 candidate: 6-tick match key (+0.49 wins/run on 15 opponents). +len6 | TR_MOVEMENT=strafe TR_PATTERN_LEN=6 | match key = 6 ticks (Batch-1 candidate) + +# 3. Dose-response on the key-length axis. +len16 | TR_MOVEMENT=strafe TR_PATTERN_LEN=16 | match key = 16 ticks + +# 4. STRUCTURAL POSITIVE CONTROL: bearing-invariant radial knob; cannot change +# the real aim direction (job j99), so a hits/wins gain here is an artifact. +rad_offset | TR_MOVEMENT=strafe TR_PATTERN_RAD_OFFSET=-20 | structural control: aim short 20 px