SirStone/SirRoboGarage
1
0
Fork 0
Code Issues 28 Pull Requests Actions Projects Releases Wiki Activity

TFIL heat-time + virtual pillar: live A/B (6 arms x 70 rounds) - neither change beats the pre-change mover

Browse Source 
Runs the pre-registered A/B for the two movement changes in HEAD: the
time-indexed bullet heat (TR_TFIL_HEAT_TIME, fca8993) and the removal of the
invented virtual centre pillar (d0750ab). One frozen binary from HEAD vs real
DrussGT: 6 arms x 10 runs x 7 rounds = 60 battles, 420 rounds, 0 failed.

Judged on damage/run and ROUND WINS only (hit rate and hits-taken are context):
hit rate would have inverted the verdict again - tau3 has the best pooled hit
rate of all arms (11.56%) and the fewest round wins (20/70).

RESULT (vs the reconstructed pre-change mover "old"):
  heat-time HURTS. tau3/tau5/tau9 lose 1.3-1.7 wins/run (p=0.0010-0.0125) and
  deal 22-38 less damage/run (p=0.004-0.047); tau15 is a wash on wins (p=0.64)
  and 22 damage/run lower (p=0.046). Nothing improves either metric.
  pillar removal does nothing measurable. old vs pillaoff: +5.7 damage/run
  (p=0.71), +0.5 wins/run (35 vs 30, p=0.43), 30.8 MORE damage taken/run
  without the pillar (p=0.040). The mechanism check proves the knob works
  (centre-box occupancy 0.09% -> 2.37%, p<0.0001; range 469 -> 443 px,
  p=0.0002), so this is a real behaviour change that buys nothing. At n=10 the
  pillar contrast is inside the MDE (33 damage/run, 1.2 wins/run), so this is
  not a proven regression.

Flags that the shipped default (pillar removed) should be reverted to the
TR_TFIL_PILLAR_ON behaviour; heat-time stays off.

Adds tools/ab/arms_heat_pillar.txt and tools/ab/ab_mechanism.py (per-tick
mechanism check: central-box occupancy, range distribution, live enemy-bullet
proximity) plus the captured summary/report fixtures.
This commit is contained in:
Davide Cappellini
2026-09-25 22:21:41 +02:00
parent f58d65d2e8
commit 48f38b80e7
7 changed files with 3121 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
common_libs/tests/fixtures/tfil_heat_pillar_ab_bands.txt
Vendored
+90
View File
@@ -0,0 +1,90 @@
====================================================================================================
HIT RATE BY RANGE BAND (our shots; band = shooter->target distance px at the fire tick)
session: /tmp/ab/heatshift arms: old, pillaoff, tau15, tau3, tau5, tau9 reference: old
====================================================================================================
band old pillaoff tau15 tau3 tau5 tau9
----------------------------------------------------------------------------------------------------------------------------------------------------------------------
0-100 2 1 50.0% 3 2 66.7% 2 2 100.0% 6 5 83.3% 6 3 50.0% 8 3 37.5%
100-200 21 3 14.3% 38 11 28.9% 16 3 18.8% 22 8 36.4% 31 14 45.2% 24 5 20.8%
200-300 134 30 22.4% 214 37 17.3% 103 20 19.4% 196 42 21.4% 216 39 18.1% 122 24 19.7%
300-450 3016 342 11.3% 3663 414 11.3% 1768 204 11.5% 3134 367 11.7% 2851 343 12.0% 1939 260 13.4%
450+ 4468 436 9.8% 3196 312 9.8% 6045 539 8.9% 3122 314 10.1% 3619 319 8.8% 5220 469 9.0%
ALL 7641 812 10.6% 7114 776 10.9% 7934 768 9.7% 6480 736 11.4% 6723 718 10.7% 7313 761 10.4%
PER-RUN BAND RATES (shows the spread behind the pooled numbers)
band 0-100
old n= 2 0 100
pillaoff n= 3 100 0 100
tau15 n= 2 100 100
tau3 n= 5 100 100 100 100 50
tau5 n= 5 100 100 50 0 0
tau9 n= 5 0 0 50 100 33
band 100-200
old n= 9 0 0 0 0 0 0 20 100 0
pillaoff n=10 0 0 50 0 50 43 56 0 25 0
tau15 n= 8 0 0 33 50 0 0 0 0
tau3 n= 8 33 0 0 40 33 67 0 0
tau5 n=10 38 100 50 50 0 33 0 0 60 100
tau9 n= 8 25 60 0 0 0 0 0 25
band 200-300
old n=10 15 14 17 26 0 25 60 18 40 30
pillaoff n=10 29 10 14 16 19 24 18 25 10 19
tau15 n=10 50 11 25 12 17 33 17 33 7 0
tau3 n=10 17 17 12 0 39 21 22 27 20 17
tau5 n=10 21 29 6 21 19 26 24 29 9 6
tau9 n=10 33 23 25 12 8 14 33 40 17 13
band 300-450
old n=10 11 12 13 9 12 10 9 14 11 12
pillaoff n=10 9 11 11 12 13 10 12 11 13 12
tau15 n=10 13 11 9 11 8 14 13 9 15 13
tau3 n=10 11 11 13 11 12 11 9 13 13 11
tau5 n=10 14 11 11 12 10 14 12 11 14 12
tau9 n=10 14 13 12 14 15 11 14 14 14 12
band 450+
old n=10 10 10 10 9 9 12 11 9 9 9
pillaoff n=10 8 13 13 7 10 10 11 8 9 10
tau15 n=10 8 10 10 9 9 8 9 8 9 9
tau3 n=10 8 11 11 7 9 12 11 11 12 10
tau5 n=10 10 8 9 10 10 7 9 9 8 8
tau9 n=10 9 8 10 9 8 10 10 9 8 9
band ALL
old n=10 11 11 11 9 10 11 11 11 11 10
pillaoff n=10 9 12 12 10 12 11 13 10 11 12
tau15 n=10 9 10 10 9 9 10 11 8 11 10
tau3 n=10 10 11 12 9 12 12 11 13 13 11
tau5 n=10 12 10 10 11 10 11 11 10 11 10
tau9 n=10 10 11 11 10 9 11 11 10 10 10
PER-BAND PERMUTATION TEST vs `old` (per-run rates, two-sided)
band arm d(pp) p method MCse MDE(pp)
------------------------------------------------------------------
0-100 pillaoff +16.67 1.0000 exact 0.0000 198.10
0-100 tau15 +50.00 1.0000 exact 0.0000 198.10
0-100 tau3 +40.00 0.2857 exact 0.0000 198.10
0-100 tau5 +0.00 1.0000 exact 0.0000 198.10
0-100 tau9 -13.33 0.9524 exact 0.0000 198.10
100-200 pillaoff +9.01 0.5427 exact 0.0000 43.80
100-200 tau15 -2.92 0.8765 exact 0.0000 43.80
100-200 tau3 +8.33 0.6226 exact 0.0000 43.80
100-200 tau5 +29.75 0.0880 exact 0.0000 43.80
100-200 tau9 +0.42 1.0000 exact 0.0000 43.80
200-300 pillaoff -6.15 0.3045 exact 0.0000 20.62
200-300 tau15 -3.96 0.5841 exact 0.0000 20.62
200-300 tau3 -5.29 0.4112 exact 0.0000 20.62
200-300 tau5 -5.54 0.3777 exact 0.0000 20.62
200-300 tau9 -2.57 0.6937 exact 0.0000 20.62
300-450 pillaoff -0.14 0.8270 exact 0.0000 2.03
300-450 tau15 -0.04 0.9677 exact 0.0000 2.03
300-450 tau3 +0.11 0.8729 exact 0.0000 2.03
300-450 tau5 +0.54 0.4566 exact 0.0000 2.03
300-450 tau9 +1.98 0.0069 exact 0.0000 2.03
450+ pillaoff +0.04 0.9606 exact 0.0000 1.21
450+ tau15 -0.87 0.0454 exact 0.0000 1.21
450+ tau3 +0.36 0.5427 exact 0.0000 1.21
450+ tau5 -1.05 0.0290 exact 0.0000 1.21
450+ tau9 -0.88 0.0456 exact 0.0000 1.21
ALL pillaoff +0.32 0.4681 exact 0.0000 0.76
ALL tau15 -0.95 0.0066 exact 0.0000 0.76
ALL tau3 +0.71 0.1325 exact 0.0000 0.76
ALL tau5 +0.06 0.8627 exact 0.0000 0.76
ALL tau9 -0.25 0.3258 exact 0.0000 0.76
common_libs/tests/fixtures/tfil_heat_pillar_ab_mechanism.txt
Vendored
+74
View File
@@ -0,0 +1,74 @@
====================================================================================================
MECHANISM CHECK — did the arm actually move differently?
session /tmp/ab/heatshift arms old, pillaoff, tau15, tau3, tau5, tau9 reference old
box = 144x144 px centred on the arena centre (x 328-472, y 228-372)
====================================================================================================
PER-ARM MECHANISM (mean of the per-round values, then over runs)
arm runs box% dist px p10 med p90 enB<=100px% enBmin px
-------------------------------------------------------------------------------
old 10 0.09 469.3 433 465 511 27.99 136.1
pillaoff 10 2.37 443.5 396 446 490 29.10 132.8
tau15 10 4.68 500.1 467 502 533 25.57 141.4
tau3 10 26.12 439.4 406 440 471 30.86 134.8
tau5 10 21.30 447.8 410 451 484 30.60 134.1
tau9 10 10.40 481.1 440 482 512 28.31 137.8
DISTANCE-TO-ENEMY BANDS (fraction of ticks, mean over rounds/runs, %)
arm 0-100 100-200 200-300 300-400 400+
------------------------------------------------------------
old 0.11 0.51 2.15 14.24 82.99
pillaoff 0.15 0.95 3.33 23.46 72.12
tau15 0.18 0.56 1.64 9.47 88.15
tau3 0.18 0.89 3.73 24.80 70.39
tau5 0.23 0.84 3.80 21.20 73.93
tau9 0.21 0.71 2.04 11.53 85.51
ENEMY-BULLET PROXIMITY (nearest LIVE ENEMY bullet; our own bullets are excluded because a bullet is born at its own tank, so 'any bullet' is dominated by our just-fired shot)
arm <=50px <=100px <=150px min px enemy%
-------------------------------------------------------
old 4.76 27.99 62.48 136.1 97.04
pillaoff 4.97 29.10 64.98 132.8 97.25
tau15 4.12 25.57 58.43 141.4 97.06
tau3 6.19 30.86 61.39 134.8 97.03
tau5 5.88 30.60 62.59 134.1 97.31
tau9 4.99 28.31 60.75 137.8 97.45
ANY-BULLET PROXIMITY (per the task text: nearest live bullet, ours included; context only)
arm <=50px <=100px <=150px min px any%
-------------------------------------------------------
old 24.96 56.38 83.24 92.4 98.17
pillaoff 25.33 57.33 84.58 90.5 98.01
tau15 24.39 54.95 81.71 94.3 98.04
tau3 26.25 57.89 82.66 90.7 97.65
tau5 25.88 57.83 83.50 90.6 97.84
tau9 25.05 56.42 82.41 92.7 98.05
PER-RUN MECHANISM vs `old` (two-sided permutation on per-run means; exact when C(n,na)<=2e7)
metric arm delta p method MDE
------------------------------------------------------------------------------
central-box occupancy pillaoff +2.29 0.0000 exact 0.11%
central-box occupancy tau15 +4.60 0.0000 exact 0.11%
central-box occupancy tau3 +26.04 0.0000 exact 0.11%
central-box occupancy tau5 +21.21 0.0000 exact 0.11%
central-box occupancy tau9 +10.31 0.0000 exact 0.11%
mean dist to enemy pillaoff -25.74 0.0002 exact 20.81px
mean dist to enemy tau15 +30.82 0.0001 exact 20.81px
mean dist to enemy tau3 -29.90 0.0004 exact 20.81px
mean dist to enemy tau5 -21.44 0.0014 exact 20.81px
mean dist to enemy tau9 +11.82 0.0813 exact 20.81px
ticks enemy bullet<=100px pillaoff +1.11 0.1381 exact 1.74%
ticks enemy bullet<=100px tau15 -2.43 0.0001 exact 1.74%
ticks enemy bullet<=100px tau3 +2.87 0.0014 exact 1.74%
ticks enemy bullet<=100px tau5 +2.61 0.0005 exact 1.74%
ticks enemy bullet<=100px tau9 +0.32 0.6066 exact 1.74%
mean dist nearest enemy bullet pillaoff -3.29 0.0155 exact 3.03px
mean dist nearest enemy bullet tau15 +5.28 0.0000 exact 3.03px
mean dist nearest enemy bullet tau3 -1.24 0.3106 exact 3.03px
mean dist nearest enemy bullet tau5 -2.01 0.1130 exact 3.03px
mean dist nearest enemy bullet tau9 +1.68 0.1133 exact 3.03px
ticks ANY bullet<=100px pillaoff +0.95 0.0394 exact 1.39%
ticks ANY bullet<=100px tau15 -1.43 0.0040 exact 1.39%
ticks ANY bullet<=100px tau3 +1.51 0.0228 exact 1.39%
ticks ANY bullet<=100px tau5 +1.45 0.0147 exact 1.39%
ticks ANY bullet<=100px tau9 +0.04 0.9388 exact 1.39%
common_libs/tests/fixtures/tfil_heat_pillar_ab_report.txt
Vendored
+101
View File
@@ -0,0 +1,101 @@
# session /tmp/ab/heatshift
# commit=f58d65d2e8206cebd5b7d0a9465950dd9e6d2c28 binary_sha256=74fd010ed1ffc4bb8a8de5569f58843479e1f50164a8a3f61e9c81cef108eaef rounds=7 runs=10 conc=7 ts=2026-09-25T22:09:42+02:00
ARM SUMMARY
arm runs dmg/run dmgtk/run wins win% shots/run hitstk/run
--------------------------------------------------------------------------
old 10 287 202 35/70 50.0 783 93.5
pillaoff 10 282 232 30/70 42.9 729 95.0
tau3 10 261 299 20/70 28.6 666 106.2
tau5 10 250 294 18/70 25.7 689 105.3
tau9 10 265 231 22/70 31.4 750 99.5
tau15 10 265 185 32/70 45.7 812 89.5
PER-RUN (never just the mean)
old dmg: r1=271 r2=273 r3=255 r4=270 r5=282 r6=309 r7=338 r8=305 r9=266 r10=306
wins: r1=3/7 r2=4/7 r3=4/7 r4=2/7 r5=2/7 r6=4/7 r7=4/7 r8=4/7 r9=3/7 r10=5/7
pillaoff dmg: r1=207 r2=332 r3=263 r4=332 r5=286 r6=280 r7=244 r8=269 r9=311 r10=294
wins: r1=1/7 r2=5/7 r3=2/7 r4=4/7 r5=4/7 r6=3/7 r7=2/7 r8=2/7 r9=4/7 r10=3/7
tau3 dmg: r1=213 r2=296 r3=182 r4=285 r5=283 r6=220 r7=290 r8=304 r9=245 r10=293
wins: r1=1/7 r2=3/7 r3=1/7 r4=3/7 r5=3/7 r6=1/7 r7=4/7 r8=1/7 r9=2/7 r10=1/7
tau5 dmg: r1=272 r2=235 r3=241 r4=236 r5=281 r6=295 r7=248 r8=243 r9=218 r10=225
wins: r1=2/7 r2=2/7 r3=2/7 r4=1/7 r5=3/7 r6=1/7 r7=2/7 r8=2/7 r9=2/7 r10=1/7
tau9 dmg: r1=295 r2=283 r3=275 r4=239 r5=268 r6=258 r7=280 r8=262 r9=228 r10=266
wins: r1=3/7 r2=3/7 r3=2/7 r4=2/7 r5=2/7 r6=3/7 r7=2/7 r8=3/7 r9=1/7 r10=1/7
tau15 dmg: r1=266 r2=264 r3=258 r4=239 r5=278 r6=291 r7=230 r8=286 r9=283 r10=261
wins: r1=3/7 r2=4/7 r3=3/7 r4=2/7 r5=4/7 r6=4/7 r7=2/7 r8=4/7 r9=4/7 r10=2/7
PAIRWISE PERMUTATION TEST (per-run values) + MANN-WHITNEY CROSS-CHECK
permutation: exact when C(n,na) <= 20,000,000; otherwise Monte-Carlo 1,000,000 draws, seed=0x5eed5eed, p = (cnt+1)/(B+1), se = sqrt(p(1-p)/(B+1))
metric A B diff(A-B) perm p method MC se MW p MW U
-------------------------------------------------------------------------------------------------
dmg/run old pillaoff +5.658 0.7096 exact - 0.8501 47.0
round wins old pillaoff +0.500 0.4317 exact - 0.3631 38.0
dmg/run old tau3 +26.343 0.1137 exact - 0.3075 36.0
round wins old tau3 +1.500 0.0125 exact - 0.0101 16.5
dmg/run old tau5 +37.893 0.0042 exact - 0.0091 15.0
round wins old tau5 +1.700 0.0010 exact - 0.0014 9.0
dmg/run old tau9 +22.056 0.0468 exact - 0.1041 28.0
round wins old tau9 +1.300 0.0097 exact - 0.0087 16.0
dmg/run old tau15 +22.039 0.0460 exact - 0.1041 28.0
round wins old tau15 +0.300 0.6369 exact - 0.5127 41.5
dmg/run pillaoff tau3 +20.686 0.2721 exact - 0.4727 40.0
round wins pillaoff tau3 +1.000 0.1149 exact - 0.0869 27.5
dmg/run pillaoff tau5 +32.236 0.0429 exact - 0.0539 24.0
round wins pillaoff tau5 +1.200 0.0262 exact - 0.0254 21.5
dmg/run pillaoff tau9 +16.398 0.2543 exact - 0.1859 32.0
round wins pillaoff tau9 +0.800 0.1548 exact - 0.1461 31.0
dmg/run pillaoff tau15 +16.382 0.2531 exact - 0.1859 32.0
round wins pillaoff tau15 -0.200 0.8378 exact - 0.7204 45.0
dmg/run tau3 tau5 +11.550 0.4674 exact - 0.3447 37.0
round wins tau3 tau5 +0.200 0.8121 exact - 0.9042 48.0
dmg/run tau3 tau9 -4.287 0.7791 exact - 0.5708 42.0
round wins tau3 tau9 -0.200 0.8197 exact - 0.6047 43.0
dmg/run tau3 tau15 -4.304 0.7776 exact - 0.6232 43.0
round wins tau3 tau15 -1.200 0.0356 exact - 0.0260 21.0
dmg/run tau5 tau9 -15.837 0.1359 exact - 0.1859 32.0
round wins tau5 tau9 -0.400 0.3567 exact - 0.2333 35.0
dmg/run tau5 tau15 -15.854 0.1337 exact - 0.1620 31.0
round wins tau5 tau15 -1.400 0.0034 exact - 0.0034 13.0
dmg/run tau9 tau15 -0.017 0.9984 exact - 1.0000 50.0
round wins tau9 tau15 -1.000 0.0356 exact - 0.0298 22.0
MINIMUM DETECTABLE EFFECT (two-sample, alpha=0.05 two-sided, 80% power; MDE = 2.8016*sd*sqrt(2/n))
metric n/arm sd(control) MDE(abs) MDE vs control mean
----------------------------------------------------------------
dmg/run 10 25.982 32.553 11.3% of 287.5
round wins 10 0.972 1.218 34.8% of 3.5
ROUND-LEVEL TEST (pooled rounds, Fisher exact) vs `old` — ANTI-CONSERVATIVE: rounds cluster within runs
arm ref wins arm wins p
----------------------------------------------
pillaoff 35/70 30/70 0.4980
tau3 35/70 20/70 0.0150
tau5 35/70 18/70 0.0051
tau9 35/70 22/70 0.0386
tau15 35/70 32/70 0.7352
LIVENESS (arm env applied in the bot's own boot report)
old OK (10/10 runs: TR_TFIL_PILLAR_ON=1 TR_TFIL_HEAT_TIME=0 applied)
pillaoff OK (10/10 runs: no arm env; report present)
tau3 OK (10/10 runs: TR_TFIL_HEAT_TIME=1 TR_TFIL_HEAT_TAU=3 applied)
tau5 OK (10/10 runs: TR_TFIL_HEAT_TIME=1 TR_TFIL_HEAT_TAU=5 applied)
tau9 OK (10/10 runs: TR_TFIL_HEAT_TIME=1 TR_TFIL_HEAT_TAU=9 applied)
tau15 OK (10/10 runs: TR_TFIL_HEAT_TIME=1 TR_TFIL_HEAT_TAU=15 applied)
[bb] APPLIED-SHIFT CHECK (from bot stdout; needs TR_BITBRAIN_LOG=1). A provably-zero placebo emits ZERO [bb] lines.
arm runs w/log lines min max zeros
old 0/10 0 - - -
pillaoff 0/10 0 - - -
tau3 0/10 0 - - -
tau5 0/10 0 - - -
tau9 0/10 0 - - -
tau15 0/10 0 - - -
ROUND-WIN ATTRIBUTION (events primary; score tie-break for mutual-kill / timeout rounds)
old wins==firstPlaces 10/10 runs OK; single-death rounds agree with score 70/70 (0 tie-broken)
pillaoff wins==firstPlaces 10/10 runs OK; single-death rounds agree with score 70/70 (0 tie-broken)
tau3 wins==firstPlaces 10/10 runs OK; single-death rounds agree with score 69/69 (1 tie-broken)
tau5 wins==firstPlaces 10/10 runs OK; single-death rounds agree with score 69/69 (1 tie-broken)
tau9 wins==firstPlaces 10/10 runs OK; single-death rounds agree with score 70/70 (0 tie-broken)
tau15 wins==firstPlaces 10/10 runs OK; single-death rounds agree with score 69/69 (1 tie-broken)
common_libs/tests/fixtures/tfil_heat_pillar_ab_results.json
Vendored
+2170
View File
File diff suppressed because it is too large Load Diff