SirStone/SirRoboGarage
1
0
Fork 0
Code Issues 28 Pull Requests Actions Projects Releases Wiki Activity
Files
a82c864c6075d8325589681bae23b371e3c4f1b5
SirRoboGarage/common_libs/tests/prediction_quality_results.txt
T
SirStone a82c864c60 bitbrain campaign phase 0: offline prediction-quality ruler and the bar 
New harness (common_libs/gun_harness/prediction_quality.nim +
common_libs/tests/run_prediction_quality.nim): per-gun single-tick aim error in
degrees against the true continuous interception point on the recorded
live-vs-real-DrussGT corpus (/tmp/tfil_ab2/out, 70 runs, 899607 ticks), per
range band, with the hit-probability proxy mean(|err|<=atan(18/range)).
Validated: recorded hits separate from misses 13.34x px (reference 11.59x),
perfect-oracle max |err| = 0, correct ordering on synthetic ground truth, two
full runs byte-identical. Fixed a wrap180 bug (Nim float mod keeps the dividend
sign) that inflated the negative error tail.

Bar (mean|err| deg [hitProxy] at 450+): Pattern 16.19 [0.077], naive-linear
22.86 [0.054], TMHorizon 16.20 [0.076], BitBrain 16.20 [0.077], static HeadOn
12.33 [0.098], oracle 0 [1.0]. Lead-gain sweep on Pattern is a dead end (1.0
wins every band). Naive-linear applies ~1.8x Pattern's lead but carries no more
lead information (corr 0.178 vs 0.165) and is strictly worse. Ledger:
docs/bitbrain_campaign.md. All verdicts remain live-only.
2026-09-24 23:40:56 +02:00

147 lines
11 KiB
Plaintext
Raw Blame History

========================================================================================================================
OFFLINE PREDICTION QUALITY -- per-gun single-tick aim error vs the true interception point
========================================================================================================================
corpus : /tmp/tfil_ab2/out
ruler : continuous (physically exact)
runs : 70
recorded ticks: 899607
tick x bin : 3598428
wall time : 406.88s (0.1131 ms per tick-bin)
per-arm speed : 0.1131 s per 1000 tick-bins per arm
NOTE: offline OPEN-LOOP prediction quality only. No win/damage/survival claim.
========================================================================================================================
VALIDATION -- the ruler must pass ALL of these before any number below is trusted
========================================================================================================================
1. recorded shots (OUR actual server-fired bearings vs the SAME interception solve):
ruler=continuous hits n=5480 mean|err|= 1.360 deg / 10.5 px | misses n=48304 mean|err|= 16.597 deg / 140.3 px | separation 12.20x deg / 13.34x px -> OK
ruler=integer hits n=5480 mean|err|= 1.478 deg / 11.4 px | misses n=48304 mean|err|= 16.724 deg / 141.3 px | separation 11.32x deg / 12.43x px -> OK
2. perfect-oracle gun max |err| over all tick-bins = 0.000000 deg -> OK
3. HeadOn (static LOS) mean|err| = 13.217 deg vs Pattern 16.609 / TMHorizon 16.627 / BitBrain 16.635
-> UNEXPECTED: a predictive gun is worse than static LOS
NaiveLinear mean|err| = 22.086 deg (over-leads; see the lead-gain sweep for why a larger
lead *response* does not mean a smaller angular error)
4. determinism: run twice and diff stdout (see fixture; verified separately).
========================================================================================================================
THE BAR -- per-band mean ABSOLUTE angular aim error (deg), RMSE, sign, hit-proxy
========================================================================================================================
hitProxy = fraction of tick-bins with |err| <= atan(18/range) (the angular half-width of the target disc).
arm band n meanAbs rmse signed hitProxy maxAbs
----------------------------------------------------------------------------------
Oracle 0-100 4423 0.000 0.000 0.000 1.0000 0.00
Oracle 100-200 24908 0.000 0.000 0.000 1.0000 0.00
Oracle 200-300 74215 0.000 0.000 0.000 1.0000 0.00
Oracle 300-450 1119777 0.000 0.000 0.000 1.0000 0.00
Oracle 450+ 2311323 0.000 0.000 0.000 1.0000 0.00
(Oracle: 63782 tick-bins had no valid interception)
OracleQuant 0-100 4423 1.103 1.498 0.078 1.0000 6.00
OracleQuant 100-200 24908 0.674 0.911 -0.002 1.0000 4.07
OracleQuant 200-300 74215 0.472 0.624 -0.008 1.0000 2.44
OracleQuant 300-450 1119777 0.360 0.470 -0.010 1.0000 1.62
OracleQuant 450+ 2311323 0.288 0.376 0.010 1.0000 1.23
(OracleQuant: 63782 tick-bins had no valid interception)
HeadOn 0-100 4423 19.619 23.254 -1.279 0.3423 46.28
HeadOn 100-200 24908 19.982 23.151 -0.462 0.1724 46.62
HeadOn 200-300 74215 17.341 20.521 0.481 0.1330 46.33
HeadOn 300-450 1119777 14.607 17.606 0.690 0.1049 46.38
HeadOn 450+ 2311323 12.326 15.017 -0.263 0.0984 45.49
(HeadOn: 63782 tick-bins had no valid interception)
Pattern 0-100 4423 10.555 14.796 -0.404 0.6993 64.72
Pattern 100-200 24908 14.745 19.510 1.276 0.3418 76.63
Pattern 200-300 74215 16.610 21.174 1.350 0.1850 81.45
Pattern 300-450 1119777 17.531 21.838 0.948 0.1036 85.65
Pattern 450+ 2311323 16.193 20.021 -0.642 0.0767 79.92
(Pattern: 63782 tick-bins had no valid interception)
PatternGain1.5 0-100 4423 13.764 18.522 0.034 0.6093 79.58
PatternGain1.5 100-200 24908 19.640 25.124 2.144 0.2025 97.08
PatternGain1.5 200-300 74215 22.260 27.672 1.784 0.1024 103.15
PatternGain1.5 300-450 1119777 23.279 28.548 1.077 0.0627 111.53
PatternGain1.5 450+ 2311323 21.245 26.062 -0.832 0.0542 104.02
(PatternGain1.5: 63782 tick-bins had no valid interception)
PatternGain2.0 0-100 4423 20.111 25.637 0.471 0.4047 95.58
PatternGain2.0 100-200 24908 26.782 33.175 3.013 0.1487 118.82
PatternGain2.0 200-300 74215 29.473 35.856 2.219 0.0776 126.59
PatternGain2.0 300-450 1119777 29.955 36.371 1.207 0.0488 137.41
PatternGain2.0 450+ 2311323 26.997 32.935 -1.021 0.0425 128.12
(PatternGain2.0: 63782 tick-bins had no valid interception)
PatternGain3.0 0-100 4423 34.772 43.079 1.346 0.2720 141.72
PatternGain3.0 100-200 24908 42.972 51.921 4.751 0.0978 162.72
PatternGain3.0 200-300 74215 45.232 54.158 3.088 0.0507 173.45
PatternGain3.0 300-450 1119777 44.274 53.364 1.462 0.0333 179.99
PatternGain3.0 450+ 2311323 39.271 47.718 -1.400 0.0293 177.73
(PatternGain3.0: 63782 tick-bins had no valid interception)
NaiveLinear 0-100 4423 17.924 35.009 -0.098 0.6505 178.26
NaiveLinear 100-200 24908 14.629 24.037 -0.445 0.3879 179.65
NaiveLinear 200-300 74215 17.572 24.479 1.030 0.1924 179.78
NaiveLinear 300-450 1119777 20.976 26.164 1.096 0.0995 179.64
NaiveLinear 450+ 2311323 22.857 27.679 -0.477 0.0537 179.98
(NaiveLinear: 63782 tick-bins had no valid interception)
TMHorizon 0-100 4423 10.681 14.782 -1.233 0.6955 66.72
TMHorizon 100-200 24908 14.841 19.526 0.814 0.3418 78.63
TMHorizon 200-300 74215 16.644 21.142 1.091 0.1786 84.45
TMHorizon 300-450 1119777 17.572 21.878 0.691 0.0996 84.01
TMHorizon 450+ 2311323 16.199 20.025 -0.687 0.0757 81.19
(TMHorizon: 63782 tick-bins had no valid interception)
BitBrain 0-100 4423 11.118 15.268 -1.507 0.6810 64.72
BitBrain 100-200 24908 15.107 19.782 1.289 0.3241 76.63
BitBrain 200-300 74215 16.838 21.426 1.341 0.1747 105.04
BitBrain 300-450 1119777 17.575 21.894 0.933 0.1025 100.98
BitBrain 450+ 2311323 16.200 20.032 -0.647 0.0767 96.38
(BitBrain: 63782 tick-bins had no valid interception)
========================================================================================================================
HEADROOM -- the direct answer: how far each arm is from the oracle ceiling, per band
========================================================================================================================
band Pattern n Pattern|err| Pattern hpx Oracle hpx headroom pp naive hpx TMHoriz hpx BitBrain hpx
---------------------------------------------------------------------------------------------------------------
0-100 4423 10.555 0.6993 1.0000 0.3007 0.6505 0.6955 0.6810
100-200 24908 14.745 0.3418 1.0000 0.6582 0.3879 0.3418 0.3241
200-300 74215 16.610 0.1850 1.0000 0.8150 0.1924 0.1786 0.1747
300-450 1119777 17.531 0.1036 1.0000 0.8964 0.0995 0.0996 0.1025
450+ 2311323 16.193 0.0767 1.0000 0.9233 0.0537 0.0757 0.0767
hitProxy = fraction of tick-bins aimed within atan(18/range) of the true interception point.
headroom pp = oracle hitProxy - Pattern hitProxy = the absolute hit-probability points available
to a perfect predictor (the campaign is playing for a slice of this).
band OracleQuant hpx integer-solve coarseness
---------------------------------------------------
0-100 1.0000 1.103 deg mean |err|
100-200 1.0000 0.674 deg mean |err|
200-300 1.0000 0.472 deg mean |err|
300-450 1.0000 0.360 deg mean |err|
450+ 1.0000 0.288 deg mean |err|
(OracleQuant aims at the analyze_lead_capture_by_range.py integer-tick intercept and is scored
on the active ruler. On the continuous ruler it measures how much of a gun's 'error' the coarse
solve itself would produce; on the integer ruler it is identically zero.)
========================================================================================================================
LEAD-GAIN SWEEP ON PATTERN -- multiply Pattern's lead (deg over LOS) by a constant
========================================================================================================================
band gain=1.0 gain=1.5 gain=2.0 gain=3.0 best-gain
------------------------------------------------------------------
0-100 10.555 13.764 20.111 34.772 1.0 (10.555)
100-200 14.745 19.640 26.782 42.972 1.0 (14.745)
200-300 16.610 22.260 29.473 45.232 1.0 (16.610)
300-450 17.531 23.279 29.955 44.274 1.0 (17.531)
450+ 16.193 21.245 26.997 39.271 1.0 (16.193)
========================================================================================================================
LEAD INFORMATIVENESS -- capture slope (regression of applied lead on required lead) and lead correlation
========================================================================================================================
capture slope is job-95's metric (1.0 = perfect proportional response). corr is the Pearson
correlation of the arm's lead with the REQUIRED lead: a large slope on an uncorrelated lead is
just amplified noise. This is the table that resolves the 'naive-linear captures 2x the lead but
hits less' tension.
band HO |err| HO |req| Pat|req| Pat cap Pat corr Lin cap Lin corr TMH cap TMH corr BB cap BB corr
-----------------------------------------------------------------------------------------------------------------------
0-100 19.619 19.619 19.619 0.649 0.774 0.573 0.362 0.668 0.776 0.701 0.768
100-200 19.982 19.982 19.982 0.553 0.612 0.592 0.523 0.560 0.614 0.570 0.611
200-300 17.341 17.341 17.341 0.449 0.457 0.519 0.429 0.452 0.460 0.459 0.457
300-450 14.607 14.607 14.607 0.278 0.266 0.476 0.324 0.273 0.262 0.280 0.266
450+ 12.326 12.326 12.326 0.175 0.165 0.310 0.178 0.174 0.164 0.175 0.165
Reference in New Issue View Git Blame Copy Permalink