SirStone/SirRoboGarage
1
0
Fork 0
Code Issues 28 Pull Requests Actions Projects Releases Wiki Activity
Files
5fe28574abb3af43af8e0bd4e86d3f8f14a54f21
SirRoboGarage/common_libs/tests/prediction_quality_results.txt
T
SirStone c305ef4212 BitBrain campaign phase 1: the missing gain sweep and BitBrain as a lead-gain corrector 
Task A — the gain region Phase 0 never covered (gain < 1). Extend the
prediction-quality ruler with gain 0.25/0.50/0.75 arms and a fixed causal
per-band arm. Full 70-run result: the hitProxy-argmax curve is
[1.00, 1.00, 1.00, 0.00, 0.00] — Pattern below 300 px, HeadOn above —
worth +0.13 pp at 300-450 and +2.16 pp at 450+ (0.0767 -> 0.0984). Lead
correlation is identical for every g>0 (Pearson is scale-invariant), so a
shrinking gain adds no lead information; and the least-squares optimum
[1,1,1,1,0.25,0.25] diverges from the hitProxy optimum because Pattern's
lead errors are bimodal.

Task B — rebuild guns/bitbrain_gun.nim as a lead-gain corrector:
aim = LOS + gain*(patternAim - LOS), gain learned online per range band by
ranking candidate gains on the hit-probability proxy (the observed lead
label via tmhObservedAt), gated to range >= 300 px. ADE+SBC output removed.
Offline (70 runs): Pattern below 300 px, +0.37 pp at 300-450, +1.90 pp at
450+ (hitProxy 0.0957 vs 0.0767), matching the fixed rule to within 0.27 pp
at 450+ and exceeding it at 300-450. ~0.0007 ms/tick marginal (old gun
~0.114 ms/tick). Default off; rack membership, env report and guard tests
(bitbrain 32, registration 13, rack 48, tm_pattern 20, env_report) unchanged
and green.

Ledger: docs/bitbrain_campaign.md Phase 1, including the three on-file
negatives and the causal-shippability note. No live claim.
2026-09-25 00:10:19 +02:00

216 lines
17 KiB
Plaintext
Raw Blame History

========================================================================================================================
OFFLINE PREDICTION QUALITY -- per-gun single-tick aim error vs the true interception point
========================================================================================================================
corpus : /tmp/tfil_ab2/out
ruler : continuous (physically exact)
runs : 70
recorded ticks: 899607
tick x bin : 3598428
wall time : 271.23s (0.0754 ms per tick-bin)
per-arm speed : 0.0754 s per 1000 tick-bins per arm
NOTE: offline OPEN-LOOP prediction quality only. No win/damage/survival claim.
========================================================================================================================
VALIDATION -- the ruler must pass ALL of these before any number below is trusted
========================================================================================================================
1. recorded shots (OUR actual server-fired bearings vs the SAME interception solve):
ruler=continuous hits n=5480 mean|err|= 1.360 deg / 10.5 px | misses n=48304 mean|err|= 16.597 deg / 140.3 px | separation 12.20x deg / 13.34x px -> OK
ruler=integer hits n=5480 mean|err|= 1.478 deg / 11.4 px | misses n=48304 mean|err|= 16.724 deg / 141.3 px | separation 11.32x deg / 12.43x px -> OK
2. perfect-oracle gun max |err| over all tick-bins = 0.000000 deg -> OK
3. HeadOn (static LOS) mean|err| = 13.217 deg vs Pattern 16.609 / TMHorizon 16.627 / BitBrain 13.627
-> UNEXPECTED: a predictive gun is worse than static LOS
NaiveLinear mean|err| = 22.086 deg (over-leads; see the lead-gain sweep for why a larger
lead *response* does not mean a smaller angular error)
4. determinism: run twice and diff stdout (see fixture; verified separately).
========================================================================================================================
THE BAR -- per-band mean ABSOLUTE angular aim error (deg), RMSE, sign, hit-proxy
========================================================================================================================
hitProxy = fraction of tick-bins with |err| <= atan(18/range) (the angular half-width of the target disc).
arm band n meanAbs rmse signed hitProxy maxAbs
----------------------------------------------------------------------------------
Oracle 0-100 4423 0.000 0.000 0.000 1.0000 0.00
Oracle 100-200 24908 0.000 0.000 0.000 1.0000 0.00
Oracle 200-300 74215 0.000 0.000 0.000 1.0000 0.00
Oracle 300-450 1119777 0.000 0.000 0.000 1.0000 0.00
Oracle 450+ 2311323 0.000 0.000 0.000 1.0000 0.00
(Oracle: 63782 tick-bins had no valid interception)
OracleQuant 0-100 4423 1.103 1.498 0.078 1.0000 6.00
OracleQuant 100-200 24908 0.674 0.911 -0.002 1.0000 4.07
OracleQuant 200-300 74215 0.472 0.624 -0.008 1.0000 2.44
OracleQuant 300-450 1119777 0.360 0.470 -0.010 1.0000 1.62
OracleQuant 450+ 2311323 0.288 0.376 0.010 1.0000 1.23
(OracleQuant: 63782 tick-bins had no valid interception)
HeadOn 0-100 4423 19.619 23.254 -1.279 0.3423 46.28
HeadOn 100-200 24908 19.982 23.151 -0.462 0.1724 46.62
HeadOn 200-300 74215 17.341 20.521 0.481 0.1330 46.33
HeadOn 300-450 1119777 14.607 17.606 0.690 0.1049 46.38
HeadOn 450+ 2311323 12.326 15.017 -0.263 0.0984 45.49
(HeadOn: 63782 tick-bins had no valid interception)
Pattern 0-100 4423 10.555 14.796 -0.404 0.6993 64.72
Pattern 100-200 24908 14.745 19.510 1.276 0.3418 76.63
Pattern 200-300 74215 16.610 21.174 1.350 0.1850 81.45
Pattern 300-450 1119777 17.531 21.838 0.948 0.1036 85.65
Pattern 450+ 2311323 16.193 20.021 -0.642 0.0767 79.92
(Pattern: 63782 tick-bins had no valid interception)
PatternGain1.5 0-100 4423 13.764 18.522 0.034 0.6093 79.58
PatternGain1.5 100-200 24908 19.640 25.124 2.144 0.2025 97.08
PatternGain1.5 200-300 74215 22.260 27.672 1.784 0.1024 103.15
PatternGain1.5 300-450 1119777 23.279 28.548 1.077 0.0627 111.53
PatternGain1.5 450+ 2311323 21.245 26.062 -0.832 0.0542 104.02
(PatternGain1.5: 63782 tick-bins had no valid interception)
PatternGain2.0 0-100 4423 20.111 25.637 0.471 0.4047 95.58
PatternGain2.0 100-200 24908 26.782 33.175 3.013 0.1487 118.82
PatternGain2.0 200-300 74215 29.473 35.856 2.219 0.0776 126.59
PatternGain2.0 300-450 1119777 29.955 36.371 1.207 0.0488 137.41
PatternGain2.0 450+ 2311323 26.997 32.935 -1.021 0.0425 128.12
(PatternGain2.0: 63782 tick-bins had no valid interception)
PatternGain3.0 0-100 4423 34.772 43.079 1.346 0.2720 141.72
PatternGain3.0 100-200 24908 42.972 51.921 4.751 0.0978 162.72
PatternGain3.0 200-300 74215 45.232 54.158 3.088 0.0507 173.45
PatternGain3.0 300-450 1119777 44.274 53.364 1.462 0.0333 179.99
PatternGain3.0 450+ 2311323 39.271 47.718 -1.400 0.0293 177.73
(PatternGain3.0: 63782 tick-bins had no valid interception)
NaiveLinear 0-100 4423 17.924 35.009 -0.098 0.6505 178.26
NaiveLinear 100-200 24908 14.629 24.037 -0.445 0.3879 179.65
NaiveLinear 200-300 74215 17.572 24.479 1.030 0.1924 179.78
NaiveLinear 300-450 1119777 20.976 26.164 1.096 0.0995 179.64
NaiveLinear 450+ 2311323 22.857 27.679 -0.477 0.0537 179.98
(NaiveLinear: 63782 tick-bins had no valid interception)
TMHorizon 0-100 4423 10.681 14.782 -1.233 0.6955 66.72
TMHorizon 100-200 24908 14.841 19.526 0.814 0.3418 78.63
TMHorizon 200-300 74215 16.644 21.142 1.091 0.1786 84.45
TMHorizon 300-450 1119777 17.572 21.878 0.691 0.0996 84.01
TMHorizon 450+ 2311323 16.199 20.025 -0.687 0.0757 81.19
(TMHorizon: 63782 tick-bins had no valid interception)
BitBrain 0-100 4423 10.555 14.796 -0.404 0.6993 64.72
BitBrain 100-200 24908 14.745 19.510 1.276 0.3418 76.63
BitBrain 200-300 74215 16.610 21.174 1.350 0.1850 81.45
BitBrain 300-450 1119777 15.518 19.170 0.773 0.1073 85.65
BitBrain 450+ 2311323 12.608 15.431 -0.351 0.0957 79.92
(BitBrain: 63782 tick-bins had no valid interception)
PatternGain0.25 0-100 4423 16.984 19.724 -1.060 0.3914 46.00
PatternGain0.25 100-200 24908 17.791 20.374 -0.027 0.1661 51.44
PatternGain0.25 200-300 74215 15.950 18.761 0.698 0.1276 53.22
PatternGain0.25 300-450 1119777 14.131 16.972 0.755 0.1025 52.64
PatternGain0.25 450+ 2311323 12.261 14.888 -0.358 0.0931 51.24
(PatternGain0.25: 63782 tick-bins had no valid interception)
PatternGain0.50 0-100 4423 14.583 16.877 -0.841 0.4689 50.78
PatternGain0.50 100-200 24908 16.103 18.679 0.407 0.1673 58.27
PatternGain0.50 200-300 74215 15.316 18.261 0.915 0.1366 62.44
PatternGain0.50 300-450 1119777 14.491 17.566 0.819 0.1003 61.45
PatternGain0.50 450+ 2311323 12.938 15.799 -0.453 0.0883 59.63
(PatternGain0.50: 63782 tick-bins had no valid interception)
PatternGain0.75 0-100 4423 12.356 15.103 -0.622 0.6396 57.28
PatternGain0.75 100-200 24908 14.965 18.369 0.841 0.2337 66.41
PatternGain0.75 200-300 74215 15.528 19.120 1.133 0.1389 71.95
PatternGain0.75 300-450 1119777 15.639 19.275 0.884 0.0952 72.71
PatternGain0.75 450+ 2311323 14.280 17.588 -0.548 0.0813 68.72
(PatternGain0.75: 63782 tick-bins had no valid interception)
PatternBandGain 0-100 4423 10.555 14.796 -0.404 0.6993 64.72
PatternBandGain 100-200 24908 14.745 19.510 1.276 0.3418 76.63
PatternBandGain 200-300 74215 16.610 21.174 1.350 0.1850 81.45
PatternBandGain 300-450 1119777 14.607 17.606 0.690 0.1049 46.38
PatternBandGain 450+ 2311323 12.326 15.017 -0.263 0.0984 45.49
(PatternBandGain: 63782 tick-bins had no valid interception)
========================================================================================================================
HEADROOM -- the direct answer: how far each arm is from the oracle ceiling, per band
========================================================================================================================
band Pattern n Pattern|err| Pattern hpx Oracle hpx headroom pp naive hpx TMHoriz hpx BitBrain hpx
---------------------------------------------------------------------------------------------------------------
0-100 4423 10.555 0.6993 1.0000 0.3007 0.6505 0.6955 0.6993
100-200 24908 14.745 0.3418 1.0000 0.6582 0.3879 0.3418 0.3418
200-300 74215 16.610 0.1850 1.0000 0.8150 0.1924 0.1786 0.1850
300-450 1119777 17.531 0.1036 1.0000 0.8964 0.0995 0.0996 0.1073
450+ 2311323 16.193 0.0767 1.0000 0.9233 0.0537 0.0757 0.0957
hitProxy = fraction of tick-bins aimed within atan(18/range) of the true interception point.
headroom pp = oracle hitProxy - Pattern hitProxy = the absolute hit-probability points available
to a perfect predictor (the campaign is playing for a slice of this).
band OracleQuant hpx integer-solve coarseness
---------------------------------------------------
0-100 1.0000 1.103 deg mean |err|
100-200 1.0000 0.674 deg mean |err|
200-300 1.0000 0.472 deg mean |err|
300-450 1.0000 0.360 deg mean |err|
450+ 1.0000 0.288 deg mean |err|
(OracleQuant aims at the analyze_lead_capture_by_range.py integer-tick intercept and is scored
on the active ruler. On the continuous ruler it measures how much of a gun's 'error' the coarse
solve itself would produce; on the integer ruler it is identically zero.)
========================================================================================================================
LEAD-GAIN SWEEP ON PATTERN -- multiply Pattern's lead (deg over LOS) by a constant
========================================================================================================================
band gain=1.0 gain=1.5 gain=2.0 gain=3.0 best-gain
------------------------------------------------------------------
0-100 10.555 13.764 20.111 34.772 1.0 (10.555)
100-200 14.745 19.640 26.782 42.972 1.0 (14.745)
200-300 16.610 22.260 29.473 45.232 1.0 (16.610)
300-450 17.531 23.279 29.955 44.274 1.0 (17.531)
450+ 16.193 21.245 26.997 39.271 1.0 (16.193)
========================================================================================================================
PHASE 1 — THE MISSING GAIN SWEEP: Pattern lead x gain in [0.00, 1.00] (0 = HeadOn, 1 = Pattern)
========================================================================================================================
Format per cell: mean|err| deg [hitProxy]. hitProxy is the objective. gain 0.0 is HeadOn,
gain 1.0 is Pattern. The per-band gain table IS causal to APPLY (range is known at fire time,
so a per-band lookup needs no learning); its ESTIMATION from these same runs is in-sample.
band |req| deg g=0.00 [hpx] g=0.25 [hpx] g=0.50 [hpx] g=0.75 [hpx] g=1.00 [hpx] bestHpx dHpx bestErr
--------------------------------------------------------------------------------------------------------------------
0-100 19.619 19.619 [0.342] 16.984 [0.391] 14.583 [0.469] 12.356 [0.640] 10.555 [0.699] 1.00 +0.0000 1.00
100-200 19.982 19.982 [0.172] 17.791 [0.166] 16.103 [0.167] 14.965 [0.234] 14.745 [0.342] 1.00 +0.0000 1.00
200-300 17.341 17.341 [0.133] 15.950 [0.128] 15.316 [0.137] 15.528 [0.139] 16.610 [0.185] 1.00 +0.0000 0.50
300-450 14.607 14.607 [0.105] 14.131 [0.102] 14.491 [0.100] 15.639 [0.095] 17.531 [0.104] 0.00 +0.0013 0.25
450+ 12.326 12.326 [0.098] 12.261 [0.093] 12.938 [0.088] 14.280 [0.081] 16.193 [0.077] 0.00 +0.0216 0.25
OPTIMAL GAIN CURVE (hitProxy-argmax per band) and its implied hit-probability gain vs Pattern:
0-100 best gain 1.00 hitProxy 0.6993 vs Pattern 0.6993 => +0.0000 pp
100-200 best gain 1.00 hitProxy 0.3418 vs Pattern 0.3418 => +0.0000 pp
200-300 best gain 1.00 hitProxy 0.1850 vs Pattern 0.1850 => +0.0000 pp
300-450 best gain 0.00 hitProxy 0.1049 vs Pattern 0.1036 => +0.0013 pp
450+ best gain 0.00 hitProxy 0.0984 vs Pattern 0.0767 => +0.0216 pp
gain = [ 0-100->1.00 100-200->1.00 200-300->1.00 300-450->0.00 450+->0.00 ]
DIRECT COMPARISON — Pattern vs the FIXED causal per-band rule [1,1,1,0.00,0.00] vs BitBrain (learned online):
band Pattern hpx fixed-band hpx BitBrain hpx fixed-Pat pp BB-Pat pp
--------------------------------------------------------------------------------
0-100 0.6993 0.6993 0.6993 +0.0000 +0.0000
100-200 0.3418 0.3418 0.3418 +0.0000 +0.0000
200-300 0.1850 0.1850 0.1850 +0.0000 +0.0000
300-450 0.1036 0.1049 0.1073 +0.0013 +0.0037
450+ 0.0767 0.0984 0.0957 +0.0216 +0.0190
fixed-band hpx = the [1,1,1,0,0] table applied causally; it was selected in-sample.
BitBrain is learned online from labels inside each run (cold start at gain 1.0).
LEAD CORRELATION PER GAIN (Pearson of applied lead with required lead). Pearson is invariant
under positive scaling, so every g>0 column must be IDENTICAL to Pattern; g=0 has no lead and
therefore no correlation. If they match, a shrinking gain does NOT add lead information — it
only shrinks the magnitude of an uninformative signal (the gain-sweep mechanism).
band corr(g) g=0.25 g=0.50 g=0.75 g=1.00
0-100 0.774 0.774 0.774 0.774
100-200 0.612 0.612 0.612 0.612
200-300 0.457 0.457 0.457 0.457
300-450 0.266 0.266 0.266 0.266
450+ 0.165 0.165 0.165 0.165
========================================================================================================================
LEAD INFORMATIVENESS -- capture slope (regression of applied lead on required lead) and lead correlation
========================================================================================================================
capture slope is job-95's metric (1.0 = perfect proportional response). corr is the Pearson
correlation of the arm's lead with the REQUIRED lead: a large slope on an uncorrelated lead is
just amplified noise. This is the table that resolves the 'naive-linear captures 2x the lead but
hits less' tension.
band HO |err| HO |req| Pat|req| Pat cap Pat corr Lin cap Lin corr TMH cap TMH corr BB cap BB corr
-----------------------------------------------------------------------------------------------------------------------
0-100 19.619 19.619 19.619 0.649 0.774 0.573 0.362 0.668 0.776 0.649 0.774
100-200 19.982 19.982 19.982 0.553 0.612 0.592 0.523 0.560 0.614 0.553 0.612
200-300 17.341 17.341 17.341 0.449 0.457 0.519 0.429 0.452 0.460 0.449 0.457
300-450 14.607 14.607 14.607 0.278 0.266 0.476 0.324 0.273 0.262 0.148 0.213
450+ 12.326 12.326 12.326 0.175 0.165 0.310 0.178 0.174 0.164 0.036 0.101
Reference in New Issue View Git Blame Copy Permalink