BitBrain vs TMHorizon vs Pattern: live A/B on shipped TFIL (null result)
4 arms x 15 runs x 7 rounds (60 battles, 0 failed) vs real DrussGT on the
shipped TFIL default, frozen at ed25ce2. bb_id (gain 1.0 identity) is
statistically indistinguishable from shipped Pattern -> plumbing validity
check passes. No BitBrain arm beats TMHorizon or Pattern: bb_learn (the config
the owner likely ran) is the worst arm (276 dmg/run, 39/105 wins), the only
comparison at alpha=0.05 is Pattern beating it on damage. Learned gains
(>=1.0, gated >=300px) over-lead and lose 1.61pp of hit rate at 300-450px.
MDE 29.5 dmg/run, 1.235 wins/run; a 6-4-sized effect needs ~39 runs/arm.
This commit is contained in:
@@ -0,0 +1,66 @@
|
||||
====================================================================================================
|
||||
HIT RATE BY RANGE BAND (our shots; band = shooter->target distance px at the fire tick)
|
||||
session: /tmp/ab/j113_bb_vs_tmh arms: bb_id, bb_learn, pattern, tmh reference: pattern
|
||||
====================================================================================================
|
||||
band bb_id bb_learn pattern tmh
|
||||
------------------------------------------------------------------------------------------------------------------
|
||||
0-100 2 0 0.0% 3 2 66.7% 10 4 40.0% 11 5 45.5%
|
||||
100-200 30 6 20.0% 46 11 23.9% 65 13 20.0% 62 14 22.6%
|
||||
200-300 287 41 14.3% 303 50 16.5% 287 45 15.7% 298 51 17.1%
|
||||
300-450 5708 711 12.5% 5673 659 11.6% 5679 753 13.3% 5726 647 11.3%
|
||||
450+ 5489 532 9.7% 5109 440 8.6% 5076 470 9.3% 4940 494 10.0%
|
||||
ALL 11516 1290 11.2% 11134 1162 10.4% 11117 1285 11.6% 11037 1211 11.0%
|
||||
|
||||
PER-RUN BAND RATES (shows the spread behind the pooled numbers)
|
||||
band 0-100
|
||||
bb_id n= 2 0 0
|
||||
bb_learn n= 2 50 100
|
||||
pattern n= 5 50 50 33 0 50
|
||||
tmh n= 6 0 0 100 100 40 50
|
||||
band 100-200
|
||||
bb_id n=14 0 0 0 50 0 0 20 0 50 25 0 50 0 100
|
||||
bb_learn n=13 0 0 17 0 0 50 50 27 0 33 50 40 33
|
||||
pattern n=13 25 0 20 14 40 20 33 0 14 0 0 29 20
|
||||
tmh n=13 0 0 0 20 25 0 50 0 17 67 29 0 25
|
||||
band 200-300
|
||||
bb_id n=15 0 20 22 10 14 14 15 10 19 11 25 15 6 19 9
|
||||
bb_learn n=15 4 14 16 0 0 14 25 20 9 19 14 39 35 11 18
|
||||
pattern n=15 29 0 17 12 19 15 17 10 33 10 0 13 24 12 10
|
||||
tmh n=15 11 32 30 12 0 14 12 26 13 14 14 17 18 10 36
|
||||
band 300-450
|
||||
bb_id n=15 11 13 13 12 11 16 13 12 12 12 10 13 14 13 13
|
||||
bb_learn n=15 10 11 11 14 11 13 13 11 11 11 10 13 11 13 10
|
||||
pattern n=15 13 14 13 16 12 13 15 11 12 15 15 13 14 11 13
|
||||
tmh n=15 10 12 11 12 12 9 10 12 14 11 12 14 10 10 11
|
||||
band 450+
|
||||
bb_id n=15 12 9 9 10 10 8 8 8 12 9 10 10 10 11 11
|
||||
bb_learn n=15 7 8 12 8 7 9 12 10 10 8 10 8 8 7 8
|
||||
pattern n=15 11 9 9 8 11 11 10 9 9 9 6 10 8 11 10
|
||||
tmh n=15 13 9 9 11 11 10 10 10 9 14 7 12 10 8 12
|
||||
band ALL
|
||||
bb_id n=15 11 11 11 11 10 12 10 10 12 11 10 11 12 12 12
|
||||
bb_learn n=15 9 9 12 11 9 11 13 11 10 10 11 11 10 11 9
|
||||
pattern n=15 12 12 12 12 12 13 12 10 11 12 11 11 11 11 11
|
||||
tmh n=15 11 11 10 12 11 10 10 11 11 12 10 14 11 9 12
|
||||
|
||||
PER-BAND PERMUTATION TEST vs `pattern` (per-run rates, two-sided)
|
||||
band arm d(pp) p method MCse MDE(pp)
|
||||
------------------------------------------------------------------
|
||||
0-100 bb_id -36.67 0.1429 exact 0.0000 38.50
|
||||
0-100 bb_learn +38.33 0.2381 exact 0.0000 38.50
|
||||
0-100 tmh +11.67 0.6017 exact 0.0000 38.50
|
||||
100-200 bb_id +4.50 0.6598 MC/B=200,000 0.0011 14.84
|
||||
100-200 bb_learn +6.55 0.3520 exact 0.0000 14.84
|
||||
100-200 tmh +1.26 0.8610 exact 0.0000 14.84
|
||||
200-300 bb_id -0.73 0.8022 MC/B=200,000 0.0009 9.33
|
||||
200-300 bb_learn +1.13 0.7635 MC/B=200,000 0.0010 9.33
|
||||
200-300 tmh +2.59 0.4574 MC/B=200,000 0.0011 9.33
|
||||
300-450 bb_id -0.72 0.1996 MC/B=200,000 0.0009 1.46
|
||||
300-450 bb_learn -1.61 0.0035 MC/B=200,000 0.0001 1.46
|
||||
300-450 tmh -1.88 0.0010 MC/B=200,000 0.0001 1.46
|
||||
450+ bb_id +0.42 0.3927 MC/B=200,000 0.0011 1.39
|
||||
450+ bb_learn -0.57 0.2987 MC/B=200,000 0.0010 1.39
|
||||
450+ tmh +0.85 0.1511 MC/B=200,000 0.0008 1.39
|
||||
ALL bb_id -0.36 0.1921 MC/B=200,000 0.0009 0.74
|
||||
ALL bb_learn -1.13 0.0021 MC/B=200,000 0.0001 0.74
|
||||
ALL tmh -0.54 0.1397 MC/B=200,000 0.0008 0.74
|
||||
@@ -0,0 +1,69 @@
|
||||
# session /tmp/ab/j113_bb_vs_tmh
|
||||
# commit=ed25ce29ad28838eaf726137e60c036e8ed70de3 binary_sha256=0fea927b5e0e17d1f82791bb620c08f1a460c074f3d21211bddc9c75f3bf1b93 rounds=7 runs=15 conc=7 ts=2026-09-25T23:41:38+02:00
|
||||
|
||||
ARM SUMMARY
|
||||
arm runs dmg/run dmgtk/run wins win% shots/run hitstk/run
|
||||
--------------------------------------------------------------------------
|
||||
pattern 15 302 238 48/105 45.7 761 99.4
|
||||
bb_id 15 296 217 50/105 47.6 787 97.5
|
||||
bb_learn 15 276 217 39/105 37.1 761 100.3
|
||||
tmh 15 294 220 46/105 43.8 753 96.0
|
||||
|
||||
PER-RUN (never just the mean)
|
||||
pattern dmg: r1=315 r2=262 r3=299 r4=284 r5=266 r6=274 r7=300 r8=280 r9=309 r10=358 r11=273 r12=338 r13=334 r14=320 r15=321
|
||||
wins: r1=3/7 r2=2/7 r3=5/7 r4=1/7 r5=1/7 r6=4/7 r7=5/7 r8=3/7 r9=3/7 r10=4/7 r11=3/7 r12=4/7 r13=3/7 r14=3/7 r15=4/7
|
||||
bb_id dmg: r1=294 r2=279 r3=320 r4=272 r5=328 r6=305 r7=338 r8=337 r9=299 r10=328 r11=281 r12=226 r13=247 r14=297 r15=292
|
||||
wins: r1=3/7 r2=4/7 r3=4/7 r4=3/7 r5=4/7 r6=3/7 r7=4/7 r8=5/7 r9=2/7 r10=4/7 r11=3/7 r12=2/7 r13=2/7 r14=4/7 r15=3/7
|
||||
bb_learn dmg: r1=204 r2=297 r3=284 r4=272 r5=257 r6=296 r7=311 r8=295 r9=237 r10=238 r11=320 r12=304 r13=225 r14=260 r15=345
|
||||
wins: r1=2/7 r2=4/7 r3=1/7 r4=2/7 r5=3/7 r6=3/7 r7=3/7 r8=3/7 r9=1/7 r10=2/7 r11=4/7 r12=4/7 r13=1/7 r14=2/7 r15=4/7
|
||||
tmh dmg: r1=279 r2=307 r3=258 r4=307 r5=262 r6=323 r7=267 r8=260 r9=320 r10=305 r11=318 r12=335 r13=306 r14=290 r15=267
|
||||
wins: r1=2/7 r2=4/7 r3=3/7 r4=3/7 r5=3/7 r6=2/7 r7=2/7 r8=3/7 r9=5/7 r10=4/7 r11=4/7 r12=3/7 r13=2/7 r14=4/7 r15=2/7
|
||||
|
||||
PAIRWISE PERMUTATION TEST (per-run values) + MANN-WHITNEY CROSS-CHECK
|
||||
permutation: exact when C(n,na) <= 20,000,000; otherwise Monte-Carlo 1,000,000 draws, seed=0x5eed5eed, p = (cnt+1)/(B+1), se = sqrt(p(1-p)/(B+1))
|
||||
metric A B diff(A-B) perm p method MC se MW p MW U
|
||||
-------------------------------------------------------------------------------------------------
|
||||
dmg/run pattern bb_id +5.967 0.5968 MC/B=1,000,000 0.0005 0.7716 105.0
|
||||
round wins pattern bb_id -0.133 0.8647 MC/B=1,000,000 0.0003 0.8450 107.5
|
||||
dmg/run pattern bb_learn +25.894 0.0483 MC/B=1,000,000 0.0002 0.0620 67.0
|
||||
round wins pattern bb_learn +0.600 0.2197 MC/B=1,000,000 0.0004 0.1837 81.0
|
||||
dmg/run pattern tmh +8.448 0.4038 MC/B=1,000,000 0.0005 0.4306 93.0
|
||||
round wins pattern tmh +0.133 0.8671 MC/B=1,000,000 0.0003 0.6196 100.5
|
||||
dmg/run bb_id bb_learn +19.927 0.1386 MC/B=1,000,000 0.0003 0.1844 80.0
|
||||
round wins bb_id bb_learn +0.733 0.0853 MC/B=1,000,000 0.0003 0.0842 72.0
|
||||
dmg/run bb_id tmh +2.481 0.8174 MC/B=1,000,000 0.0004 0.7089 103.0
|
||||
round wins bb_id tmh +0.267 0.5574 MC/B=1,000,000 0.0005 0.4206 93.5
|
||||
dmg/run bb_learn tmh -17.446 0.1600 MC/B=1,000,000 0.0004 0.1585 78.0
|
||||
round wins bb_learn tmh -0.467 0.3019 MC/B=1,000,000 0.0005 0.3012 88.0
|
||||
|
||||
MINIMUM DETECTABLE EFFECT (two-sample, alpha=0.05 two-sided, 80% power; MDE = 2.8016*sd*sqrt(2/n))
|
||||
metric n/arm sd(control) MDE(abs) MDE vs control mean
|
||||
----------------------------------------------------------------
|
||||
dmg/run 15 28.854 29.517 9.8% of 302.2
|
||||
round wins 15 1.207 1.235 38.6% of 3.2
|
||||
|
||||
ROUND-LEVEL TEST (pooled rounds, Fisher exact) vs `pattern` — ANTI-CONSERVATIVE: rounds cluster within runs
|
||||
arm ref wins arm wins p
|
||||
----------------------------------------------
|
||||
bb_id 48/105 50/105 0.8900
|
||||
bb_learn 48/105 39/105 0.2624
|
||||
tmh 48/105 46/105 0.8897
|
||||
|
||||
LIVENESS (arm env applied in the bot's own boot report)
|
||||
pattern OK (15/15 runs: no arm env; report present)
|
||||
bb_id OK (15/15 runs: TR_RACK_PATTERN=off TR_RACK_BITBRAIN=both TR_BITBRAIN_GAINS=1.0 applied)
|
||||
bb_learn OK (15/15 runs: TR_RACK_PATTERN=off TR_RACK_BITBRAIN=both TR_BITBRAIN_GAINS=1.0,1.25,1.5,2.0 TR_BITBRAIN_MEM=decay applied)
|
||||
tmh OK (15/15 runs: TR_RACK_PATTERN=off TR_RACK_TMHORIZON=both applied)
|
||||
|
||||
[bb] APPLIED-SHIFT CHECK (from bot stdout; needs TR_BITBRAIN_LOG=1). A provably-zero placebo emits ZERO [bb] lines.
|
||||
arm runs w/log lines min max zeros
|
||||
pattern 0/15 0 - - -
|
||||
bb_id 0/15 0 - - -
|
||||
bb_learn 0/15 0 - - -
|
||||
tmh 0/15 0 - - -
|
||||
|
||||
ROUND-WIN ATTRIBUTION (events primary; score tie-break for mutual-kill / timeout rounds)
|
||||
pattern wins==firstPlaces 15/15 runs OK; single-death rounds agree with score 104/104 (1 tie-broken)
|
||||
bb_id wins==firstPlaces 15/15 runs OK; single-death rounds agree with score 104/104 (1 tie-broken)
|
||||
bb_learn wins==firstPlaces 15/15 runs OK; single-death rounds agree with score 105/105 (0 tie-broken)
|
||||
tmh wins==firstPlaces 15/15 runs OK; single-death rounds agree with score 104/104 (1 tie-broken)
|
||||
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"commit": "ed25ce29ad28838eaf726137e60c036e8ed70de3",
|
||||
"binary_sha256": "0fea927b5e0e17d1f82791bb620c08f1a460c074f3d21211bddc9c75f3bf1b93",
|
||||
"binary": "frozen/ModularBot/ModularBot_bin",
|
||||
"rounds": 7,
|
||||
"runs": 15,
|
||||
"conc": 7,
|
||||
"timestamp": "2026-09-25T23:41:38+02:00",
|
||||
"outdir": "/tmp/ab/j113_bb_vs_tmh",
|
||||
"arms": [
|
||||
{"name": "pattern", "env": "", "label": ""},
|
||||
{"name": "bb_id", "env": "TR_RACK_PATTERN=off TR_RACK_BITBRAIN=both TR_BITBRAIN_GAINS=1.0", "label": "gain 1.0 identity (validity check)"},
|
||||
{"name": "bb_learn", "env": "TR_RACK_PATTERN=off TR_RACK_BITBRAIN=both TR_BITBRAIN_GAINS=1.0,1.25,1.5,2.0 TR_BITBRAIN_MEM=decay", "label": "learned gain"},
|
||||
{"name": "tmh", "env": "TR_RACK_PATTERN=off TR_RACK_TMHORIZON=both", "label": "TMHorizon-only rack"}
|
||||
]
|
||||
}
|
||||
Reference in New Issue
Block a user