SirStone/SirRoboGarage
1
0
Fork 0
Code Issues 28 Pull Requests Actions Projects Releases Wiki Activity

TFIL: measure the commitment A/B on 490 live DrussGT rounds - no effect

Browse Source 
Runs the five-arm commitment A/B that 19bf461 only implemented. One frozen
binary (git archive 19bf461, sha256 46e7ce19...) vs the real DrussGT through
tools/robocode_shim/run_bridge_battle.sh: 7 runs x 7 rounds per arm against
real DrussGT in the pre-registered block, plus an independent replication
block (runs 8-14) - 70 battles, 490 rounds, 5 arms in parallel.

  A control (shipped)          B TR_TFIL_TILE_REPLAN=off
  C B + TR_TFIL_NO_REV=1       D TR_TFIL_TILE_REPLAN=enemy
  E B + TR_TFIL_COMMIT_TICKS=30   (TR_TFIL_COMMIT_LOG=1 on every arm)

RESULT: no arm improves damage/run or round wins vs the shipped mover. Pooled
(14 runs/arm, exact two-sided permutation test over all C(28,14) relabellings):

  arm  damage/run (delta, p)        round wins (delta, p)
  A    274.44                       2.79  (39/98)
  B    274.07 (-0.37, p=0.97)       3.07  (+0.29, p=0.66)
  C    257.34 (-17.10, p=0.16)      2.43  (-0.36, p=0.47)
  D    286.61 (+12.17, p=0.36)      3.29  (+0.50, p=0.35)
  E    265.29 (-9.15, p=0.56)       2.93  (+0.14, p=0.89)

The replication is what settles it: block 1 alone showed D at +20.99 damage
(p=0.26); block 2 put D at +3.36. Nothing replicates.

The PREMISE fails. Honouring the commitment does not reduce reversals - it
raises the reversal-pick rate from 33.9% (control) to 50.9% (B) / 57.0% (E);
the no-reversal arm C only claws part of it back (42.4%). The post-reversal
speed dip (~4.9 -> ~4.0 px/tick at +1..+2 calls) is identical in every arm, so
it is a property of turning around, not of the tile replan, and arm A - the
arm carrying the bug - has the LOWEST mean abs(speed) of all five.

Both premise premises are also caught by the metric the report is careful
about: arm C takes significantly FEWER hits (87.2 vs 96.5/run, p=0.0022) while
dealing the least damage and winning the fewest rounds - a movement change
alters hits taken, shots fired and round length at once, which is exactly why
damage and round wins are the primary metrics and hit rate is not.

Treatment verified per arm from the per-tick TR_TFIL_COMMIT_LOG: the pick
interval moves off ~5 calls only where the arm says it should (A 5.05 with
96.6% tile_self replans; B 14.77 with 0%; C 14.79; D 5.29 with 95.7%
tile_enemy; E 26.62), matching the offline fixture replay (5.07 / 14.65 /
14.65 / 4.94 / 27.57). Every arm ran.

Round wins are the final RESULTS firstPlaces, cross-checked against each
round's BotDeathEvent (35/35 and 35/35 runs agree). Damage is the server's
BulletHitBotEvent damage, attributed by numeric owner/victim id and verified
against the capture's own event counters.

measure_tfil_commit_ab.nim reads the raw run artifacts (JSONL states, events
sidecar, capture stdout, commit log) and prints the whole deliverable:
arm table, per-run values, the exact per-run permutation test, the exact
round-level hypergeometric test (flagged anti-conservative - rounds cluster
within a run), the treatment diagnostics, and the direct answer. It reproduces
either report offline from the committed summary:

  nim c -r --path:common_libs common_libs/tests/measure_tfil_commit_ab.nim \
      --from-summary common_libs/tests/fixtures/tfil_commit_ab_results_runs14.json

The raw ~200 MB of battle artifacts are not committed; the committed summary
holds every per-run value plus the arm-level diagnostics.
This commit is contained in:
Davide Cappellini
2026-09-24 20:24:34 +02:00
parent c8b2a8c6c5
commit cc11ede824
5 changed files with 5888 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
common_libs/tests/fixtures/tfil_commit_ab_report_runs14.md
Vendored
+292
View File
@@ -0,0 +1,292 @@
# TFIL commitment A/B — measured
POOLED: runs 1-7 (pre-registered primary) + runs 8-14 (independent replication; 70 battles, 490 rounds). The replication can be analysed alone with --first-run 8 --runs 14.
source commit: `19bf461`
frozen ModularBot binary sha256: `46e7ce19cc7889dacc0c5add9b29928dadb757fbc6ecbdb7765cddd7813361c0`
raw artifacts (NOT committed, live only): `/tmp/tfil_ab2/out/<arm>/run<n>.jsonl` + `...events.jsonl` and `/tmp/tfil_ab2/logs/<arm>/run<n>.out` + `...commit.jsonl`
analysed runs per arm: 14 (this file is self-consistent; re-run the tool with --runs N on the raw dir for a different block)
damage = server BulletHitBotEvent damage, attributed by numeric owner/victim id and verified against the capture's own counters; round wins = final RESULTS `firstPlaces`, cross-checked against BotDeathEvent.
## 1. Arm table (14 runs x 7 rounds vs real DrussGT, binary frozen from one source commit)
| arm | env | damage/run | damage taken/run | ROUND WINS | round win % | our shots/run | hits taken/run | our score/run | ticks/run |
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|
| A control (shipped) | `` | 274.44 | 198.95 | 39/98 | 39.8% | 826.7 | 96.5 | 463.6 | 939 |
| B honour commitment | `TR_TFIL_TILE_REPLAN=off` | 274.07 | 192.16 | 43/98 | 43.9% | 810.1 | 89.0 | 476.8 | 915 |
| C B + no-reversal | `TR_TFIL_TILE_REPLAN=off TR_TFIL_NO_REV=1` | 257.34 | 202.16 | 34/98 | 34.7% | 805.4 | 87.2 | 421.9 | 927 |
| D enemy-keyed cancel | `TR_TFIL_TILE_REPLAN=enemy` | 286.61 | 216.09 | 46/98 | 46.9% | 784.6 | 92.5 | 503.4 | 894 |
| E commit 30 ticks | `TR_TFIL_TILE_REPLAN=off TR_TFIL_COMMIT_TICKS=30` | 265.29 | 198.96 | 41/98 | 41.8% | 790.4 | 92.4 | 459.1 | 913 |
## 2. Per-run values
**A control (shipped)** ``
| run | damage/run | damage taken | wins | round-by-round | hits taken | our shots | our score | enemy score | ticks |
|---:|---:|---:|---:|---|---:|---:|---:|---:|---:|
| 1 | 245.37 | 218.70 | 2/7 | L W L L L W L | 96 | 705 | 387 | 549 | 11166 |
| 2 | 256.58 | 240.50 | 1/7 | W L L L L L L | 101 | 828 | 324 | 640 | 13516 |
| 3 | 283.54 | 197.00 | 2/7 | L L L L W L W | 86 | 788 | 423 | 524 | 12607 |
| 4 | 306.11 | 198.00 | 3/7 | L L W L W W L | 95 | 848 | 509 | 461 | 13146 |
| 5 | 267.51 | 147.60 | 3/7 | W L L W L W L | 92 | 849 | 477 | 404 | 13483 |
| 6 | 271.07 | 138.80 | 4/7 | L W W L W W L | 88 | 885 | 546 | 327 | 13895 |
| 7 | 332.48 | 206.30 | 4/7 | W W L L W L W | 112 | 912 | 559 | 418 | 14461 |
| 8 | 296.97 | 198.00 | 4/7 | W W L L W L W | 101 | 851 | 575 | 397 | 13421 |
| 9 | 305.66 | 207.40 | 4/7 | W L L W W L W | 96 | 870 | 584 | 406 | 14053 |
| 10 | 228.06 | 195.40 | 4/7 | L W L W W L W | 89 | 831 | 501 | 396 | 12967 |
| 11 | 276.63 | 184.30 | 1/7 | L L L W L L L | 100 | 865 | 348 | 575 | 13963 |
| 12 | 238.65 | 245.00 | 2/7 | W W L L L L L | 101 | 751 | 375 | 583 | 11993 |
| 13 | 249.08 | 189.20 | 3/7 | L L L L W W W | 88 | 796 | 458 | 450 | 12709 |
| 14 | 284.47 | 219.10 | 2/7 | L W L L L W L | 106 | 795 | 424 | 550 | 12720 |
**B honour commitment** `TR_TFIL_TILE_REPLAN=off`
| run | damage/run | damage taken | wins | round-by-round | hits taken | our shots | our score | enemy score | ticks |
|---:|---:|---:|---:|---|---:|---:|---:|---:|---:|
| 1 | 270.09 | 155.90 | 2/7 | L W L W L L L | 90 | 836 | 410 | 480 | 13267 |
| 2 | 263.83 | 234.60 | 1/7 | L L L L W L L | 111 | 853 | 333 | 631 | 13861 |
| 3 | 280.10 | 189.00 | 5/7 | W L W L W W W | 71 | 721 | 624 | 321 | 11326 |
| 4 | 307.40 | 139.70 | 6/7 | W W W W W W L | 75 | 821 | 707 | 206 | 13196 |
| 5 | 249.24 | 266.20 | 2/7 | L L W L L L W | 99 | 738 | 388 | 594 | 11626 |
| 6 | 275.90 | 161.30 | 4/7 | W L W W L L W | 87 | 812 | 550 | 360 | 12712 |
| 7 | 320.81 | 216.10 | 3/7 | W L L W W L L | 97 | 778 | 511 | 477 | 12428 |
| 8 | 251.62 | 193.80 | 2/7 | W W L L L L L | 81 | 736 | 376 | 519 | 11868 |
| 9 | 292.65 | 159.50 | 4/7 | L W L W W W L | 71 | 834 | 565 | 356 | 12973 |
| 10 | 218.17 | 172.30 | 3/7 | L W W L L L W | 90 | 825 | 367 | 456 | 12792 |
| 11 | 254.60 | 235.00 | 2/7 | L L W L W L L | 90 | 808 | 392 | 563 | 12801 |
| 12 | 275.82 | 172.80 | 3/7 | W W W L L L L | 92 | 880 | 481 | 437 | 13974 |
| 13 | 290.25 | 177.50 | 2/7 | W L L L L W L | 104 | 861 | 419 | 502 | 13750 |
| 14 | 286.48 | 216.60 | 4/7 | W W W L L W L | 88 | 839 | 552 | 419 | 12840 |
**C B + no-reversal** `TR_TFIL_TILE_REPLAN=off TR_TFIL_NO_REV=1`
| run | damage/run | damage taken | wins | round-by-round | hits taken | our shots | our score | enemy score | ticks |
|---:|---:|---:|---:|---|---:|---:|---:|---:|---:|
| 1 | 240.31 | 200.90 | 3/7 | W L W L L W L | 89 | 787 | 446 | 467 | 13019 |
| 2 | 298.51 | 138.00 | 3/7 | W L L L W L W | 86 | 830 | 500 | 398 | 13598 |
| 3 | 244.43 | 210.30 | 2/7 | W L L W L L L | 74 | 803 | 379 | 545 | 12935 |
| 4 | 235.60 | 244.40 | 3/7 | L W W L W L L | 80 | 789 | 434 | 501 | 12292 |
| 5 | 268.72 | 193.20 | 3/7 | W L W W L L L | 93 | 839 | 466 | 458 | 13448 |
| 6 | 263.44 | 177.00 | 3/7 | W L L L L W W | 82 | 828 | 470 | 440 | 13385 |
| 7 | 202.55 | 214.70 | 0/7 | L L L L L L L | 96 | 790 | 205 | 676 | 13037 |
| 8 | 289.38 | 208.90 | 3/7 | L L L W L W W | 88 | 756 | 495 | 458 | 12233 |
| 9 | 249.73 | 192.80 | 2/7 | W L L L L L W | 90 | 809 | 388 | 522 | 13027 |
| 10 | 259.55 | 220.60 | 2/7 | L L W W L L L | 86 | 823 | 394 | 539 | 13551 |
| 11 | 274.71 | 189.60 | 3/7 | W W L L L L W | 86 | 842 | 471 | 457 | 13419 |
| 12 | 293.64 | 231.80 | 3/7 | W W W L L L L | 98 | 757 | 504 | 503 | 12005 |
| 13 | 288.62 | 225.00 | 3/7 | L W L W W L L | 94 | 829 | 494 | 481 | 12897 |
| 14 | 193.57 | 183.10 | 1/7 | L W L L L L L | 79 | 794 | 261 | 573 | 12955 |
**D enemy-keyed cancel** `TR_TFIL_TILE_REPLAN=enemy`
| run | damage/run | damage taken | wins | round-by-round | hits taken | our shots | our score | enemy score | ticks |
|---:|---:|---:|---:|---|---:|---:|---:|---:|---:|
| 1 | 247.43 | 219.90 | 2/7 | W L W L L L L | 94 | 774 | 386 | 544 | 12775 |
| 2 | 332.70 | 202.50 | 5/7 | W W L W W L W | 104 | 864 | 624 | 350 | 13863 |
| 3 | 264.08 | 201.30 | 4/7 | L W L W L W W | 78 | 765 | 488 | 422 | 12557 |
| 4 | 276.78 | 215.00 | 1/7 | L L L L L W L | 97 | 806 | 332 | 612 | 12857 |
| 5 | 339.18 | 259.40 | 3/7 | W W L W L L L | 104 | 770 | 552 | 536 | 12453 |
| 6 | 321.08 | 161.70 | 4/7 | L L W W W L W | 89 | 809 | 601 | 351 | 12585 |
| 7 | 328.30 | 219.40 | 5/7 | W L W W W L W | 87 | 752 | 669 | 360 | 11942 |
| 8 | 250.60 | 230.60 | 4/7 | W W L L W L W | 90 | 721 | 526 | 436 | 11343 |
| 9 | 266.94 | 189.40 | 3/7 | W W L L L W L | 91 | 790 | 479 | 451 | 12759 |
| 10 | 286.03 | 199.50 | 4/7 | L W W L W W L | 90 | 839 | 560 | 402 | 13223 |
| 11 | 261.72 | 200.40 | 3/7 | W L L W W L L | 94 | 817 | 462 | 466 | 12946 |
| 12 | 218.76 | 290.60 | 1/7 | L L L W L L L | 76 | 636 | 284 | 695 | 10073 |
| 13 | 339.10 | 218.30 | 3/7 | W L W L L L W | 97 | 789 | 531 | 486 | 12430 |
| 14 | 279.86 | 217.20 | 4/7 | W W W L W L L | 104 | 852 | 554 | 421 | 13479 |
**E commit 30 ticks** `TR_TFIL_TILE_REPLAN=off TR_TFIL_COMMIT_TICKS=30`
| run | damage/run | damage taken | wins | round-by-round | hits taken | our shots | our score | enemy score | ticks |
|---:|---:|---:|---:|---|---:|---:|---:|---:|---:|
| 1 | 316.56 | 215.20 | 3/7 | W W L L W L L | 110 | 847 | 513 | 483 | 13347 |
| 2 | 316.14 | 206.00 | 3/7 | W W L W L L L | 101 | 776 | 520 | 468 | 12588 |
| 3 | 271.46 | 206.20 | 3/7 | L L W L W W L | 104 | 788 | 441 | 498 | 12580 |
| 4 | 300.07 | 195.70 | 5/7 | L W L W W W W | 87 | 754 | 596 | 352 | 12233 |
| 5 | 256.01 | 188.80 | 3/7 | W L L W W L L | 94 | 828 | 465 | 452 | 13629 |
| 6 | 277.83 | 190.80 | 3/7 | W W L W L L L | 90 | 749 | 485 | 457 | 12302 |
| 7 | 263.14 | 172.90 | 4/7 | W L W W W L L | 87 | 809 | 536 | 374 | 12614 |
| 8 | 267.62 | 202.60 | 2/7 | L L W W L L L | 98 | 833 | 406 | 532 | 13384 |
| 9 | 179.35 | 214.00 | 1/7 | L W L L L L L | 81 | 740 | 247 | 608 | 11998 |
| 10 | 345.39 | 175.40 | 5/7 | W W W W W L L | 94 | 879 | 689 | 303 | 13959 |
| 11 | 174.71 | 200.50 | 0/7 | L L L L L L L | 89 | 793 | 175 | 658 | 13467 |
| 12 | 235.76 | 222.90 | 2/7 | W L W L L L L | 91 | 756 | 367 | 556 | 12571 |
| 13 | 226.19 | 197.60 | 2/7 | W W L L L L L | 92 | 788 | 364 | 531 | 12818 |
| 14 | 283.86 | 196.90 | 5/7 | L W W L W W W | 75 | 726 | 624 | 334 | 11517 |
### Data integrity: id mapping + round-win cross-check
Round wins are taken from the final RESULTS `firstPlaces` and cross-checked against the BotDeathEvent of each round (in a TR 1v1 the round loser is the bot that died). A `WARNING` here means the two disagreed, i.e. do not trust that run.
- A/run1: id 1==ModularBot, id 2==DrussGT; of 1536 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.9% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- A/run2: id 1==ModularBot, id 2==DrussGT; of 1855 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 44.6% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- A/run3: id 1==ModularBot, id 2==DrussGT; of 1741 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.3% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- A/run4: id 2==ModularBot, id 1==DrussGT; of 1806 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 47.0% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- A/run5: id 1==ModularBot, id 2==DrussGT; of 1878 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.2% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- A/run6: id 2==ModularBot, id 1==DrussGT; of 1947 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.5% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- A/run7: id 2==ModularBot, id 1==DrussGT; of 1984 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 46.0% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- A/run8: id 1==ModularBot, id 2==DrussGT; of 1877 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.3% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- A/run9: id 1==ModularBot, id 2==DrussGT; of 1918 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.4% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- A/run10: id 1==ModularBot, id 2==DrussGT; of 1796 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 46.3% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- A/run11: id 2==ModularBot, id 1==DrussGT; of 1933 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 44.7% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- A/run12: id 1==ModularBot, id 2==DrussGT; of 1649 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.5% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- A/run13: id 2==ModularBot, id 1==DrussGT; of 1764 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.1% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- A/run14: id 1==ModularBot, id 2==DrussGT; of 1749 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.5% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- B/run1: id 2==ModularBot, id 1==DrussGT; of 1829 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.7% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- B/run2: id 1==ModularBot, id 2==DrussGT; of 1893 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.1% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- B/run3: id 2==ModularBot, id 1==DrussGT; of 1574 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.8% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- B/run4: id 2==ModularBot, id 1==DrussGT; of 1787 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.9% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- B/run5: id 1==ModularBot, id 2==DrussGT; of 1606 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 46.0% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- B/run6: id 2==ModularBot, id 1==DrussGT; of 1759 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 46.2% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- B/run7: id 2==ModularBot, id 1==DrussGT; of 1679 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 46.3% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- B/run8: id 1==ModularBot, id 2==DrussGT; of 1625 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.3% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- B/run9: id 2==ModularBot, id 1==DrussGT; of 1809 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 46.1% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- B/run10: id 1==ModularBot, id 2==DrussGT; of 1761 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 46.8% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- B/run11: id 1==ModularBot, id 2==DrussGT; of 1752 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 46.1% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- B/run12: id 1==ModularBot, id 2==DrussGT; of 1928 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.6% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- B/run13: id 1==ModularBot, id 2==DrussGT; of 1898 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.4% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- B/run14: id 1==ModularBot, id 2==DrussGT; of 1760 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 47.7% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- C/run1: id 1==ModularBot, id 2==DrussGT; of 1768 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 44.5% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- C/run2: id 1==ModularBot, id 2==DrussGT; of 1869 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 44.4% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- C/run3: id 2==ModularBot, id 1==DrussGT; of 1778 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.2% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- C/run4: id 1==ModularBot, id 2==DrussGT; of 1671 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 47.2% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- C/run5: id 2==ModularBot, id 1==DrussGT; of 1831 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.8% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- C/run6: id 1==ModularBot, id 2==DrussGT; of 1823 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.4% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- C/run7: id 1==ModularBot, id 2==DrussGT; of 1776 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 44.5% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- C/run8: id 2==ModularBot, id 1==DrussGT; of 1676 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.1% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- C/run9: id 1==ModularBot, id 2==DrussGT; of 1777 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.5% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- C/run10: id 1==ModularBot, id 2==DrussGT; of 1846 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 44.6% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- C/run11: id 1==ModularBot, id 2==DrussGT; of 1819 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 46.3% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- C/run12: id 2==ModularBot, id 1==DrussGT; of 1642 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 46.1% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- C/run13: id 1==ModularBot, id 2==DrussGT; of 1785 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 46.4% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- C/run14: id 2==ModularBot, id 1==DrussGT; of 1772 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 44.8% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- D/run1: id 1==ModularBot, id 2==DrussGT; of 1739 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 44.5% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- D/run2: id 2==ModularBot, id 1==DrussGT; of 1868 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 46.3% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- D/run3: id 1==ModularBot, id 2==DrussGT; of 1680 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.5% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- D/run4: id 1==ModularBot, id 2==DrussGT; of 1770 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.5% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- D/run5: id 1==ModularBot, id 2==DrussGT; of 1695 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.4% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- D/run6: id 2==ModularBot, id 1==DrussGT; of 1759 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 46.0% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- D/run7: id 2==ModularBot, id 1==DrussGT; of 1655 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.4% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- D/run8: id 1==ModularBot, id 2==DrussGT; of 1571 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.9% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- D/run9: id 1==ModularBot, id 2==DrussGT; of 1756 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.0% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- D/run10: id 1==ModularBot, id 2==DrussGT; of 1818 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 46.1% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- D/run11: id 2==ModularBot, id 1==DrussGT; of 1749 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 46.7% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- D/run12: id 1==ModularBot, id 2==DrussGT; of 1381 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 46.1% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- D/run13: id 1==ModularBot, id 2==DrussGT; of 1730 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.6% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- D/run14: id 2==ModularBot, id 1==DrussGT; of 1820 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 46.8% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- E/run1: id 2==ModularBot, id 1==DrussGT; of 1839 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 46.1% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- E/run2: id 2==ModularBot, id 1==DrussGT; of 1731 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 44.8% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- E/run3: id 2==ModularBot, id 1==DrussGT; of 1727 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.6% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- E/run4: id 2==ModularBot, id 1==DrussGT; of 1647 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.8% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- E/run5: id 2==ModularBot, id 1==DrussGT; of 1848 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 44.8% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- E/run6: id 2==ModularBot, id 1==DrussGT; of 1657 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.2% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- E/run7: id 2==ModularBot, id 1==DrussGT; of 1760 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 46.0% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- E/run8: id 2==ModularBot, id 1==DrussGT; of 1835 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.4% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- E/run9: id 2==ModularBot, id 1==DrussGT; of 1636 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.2% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- E/run10: id 2==ModularBot, id 1==DrussGT; of 1896 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 46.4% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- E/run11: id 1==ModularBot, id 2==DrussGT; of 1800 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 44.1% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- E/run12: id 1==ModularBot, id 2==DrussGT; of 1692 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 44.7% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- E/run13: id 1==ModularBot, id 2==DrussGT; of 1732 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.5% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- E/run14: id 1==ModularBot, id 2==DrussGT; of 1590 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.7% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
## 3. Exact two-sided permutation tests vs arm A (control = shipped mover)
Per-run test: all C(28,14)=40116600 relabellings of the 14+14 per-run values; statistic = difference of means.
| metric | A mean | arm | arm mean | delta (arm - A) | exact two-sided p | perms |
|---|---:|---|---:|---:|---:|---:|
| damage/run | 274.44 | B | 274.07 | -0.37 | 0.9720 | 40116600 |
| damage/run | 274.44 | C | 257.34 | -17.10 | 0.1559 | 40116600 |
| damage/run | 274.44 | D | 286.61 | 12.17 | 0.3567 | 40116600 |
| damage/run | 274.44 | E | 265.29 | -9.15 | 0.5574 | 40116600 |
| round wins/run | 2.79 | B | 3.07 | 0.29 | 0.6561 | 40116600 |
| round wins/run | 2.79 | C | 2.43 | -0.36 | 0.4720 | 40116600 |
| round wins/run | 2.79 | D | 3.29 | 0.50 | 0.3500 | 40116600 |
| round wins/run | 2.79 | E | 2.93 | 0.14 | 0.8865 | 40116600 |
| hits taken/run | 96.50 | B | 89.00 | -7.50 | 0.0577 | 40116600 |
| hits taken/run | 96.50 | C | 87.21 | -9.29 | 0.0022 | 40116600 |
| hits taken/run | 96.50 | D | 92.50 | -4.00 | 0.2132 | 40116600 |
| hits taken/run | 96.50 | E | 92.36 | -4.14 | 0.2078 | 40116600 |
| our shots/run | 826.71 | B | 810.14 | -16.57 | 0.4099 | 40116600 |
| our shots/run | 826.71 | C | 805.43 | -21.29 | 0.2121 | 40116600 |
| our shots/run | 826.71 | D | 784.57 | -42.14 | 0.0590 | 40116600 |
| our shots/run | 826.71 | E | 790.43 | -36.29 | 0.0675 | 40116600 |
| our score/run | 463.57 | B | 476.79 | 13.21 | 0.7283 | 40116600 |
| our score/run | 463.57 | C | 421.93 | -41.64 | 0.2270 | 40116600 |
| our score/run | 463.57 | D | 503.43 | 39.86 | 0.2919 | 40116600 |
| our score/run | 463.57 | E | 459.14 | -4.43 | 0.9224 | 40116600 |
| ticks/run | 13150.00 | B | 12815.29 | -334.71 | 0.3017 | 40116600 |
| ticks/run | 13150.00 | C | 12985.79 | -164.21 | 0.5526 | 40116600 |
| ticks/run | 13150.00 | D | 12520.36 | -629.64 | 0.0770 | 40116600 |
| ticks/run | 13150.00 | E | 12786.21 | -363.79 | 0.2331 | 40116600 |
### Round-level test (exact, anti-conservative)
Sum statistic over all rounds of the arm vs arm A only; exact hypergeometric null. ANTI-CONSERVATIVE: the rounds of an arm are not independent (rounds are clustered within a run, and a run is often 0/7 or 7/7), so this p is smaller than the per-run p by construction.
| arm | rounds won | rounds lost | all rounds pooled | exact p (round level, vs A) |
|---|---:|---:|---|---:|
| A | 39 | 59 | 39/98 | 1.0000 |
| B | 43 | 55 | 43/98 | 0.6641 |
| C | 34 | 64 | 34/98 | 0.5547 |
| D | 46 | 52 | 46/98 | 0.3872 |
| E | 41 | 57 | 41/98 | 0.8845 |
## 4. TREATMENT diagnostics (TR_TFIL_COMMIT_LOG — did the arm actually apply?)
| arm | commit calls | picks | interval (calls) | % tile_self replans | % tile_enemy | % danger | % expiry | % init | reversal-pick rate | mean abs(speed) |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| A | 183270 | 36190 | 5.05 | 96.6% | 0.0% | 0.5% | 2.6% | 0.3% | 33.9% | 4.69 |
| B | 178439 | 12029 | 14.77 | 0.0% | 0.0% | 4.8% | 94.3% | 0.9% | 50.9% | 5.37 |
| C | 180987 | 12184 | 14.79 | 0.0% | 0.0% | 4.4% | 94.7% | 0.9% | 42.4% | 5.47 |
| D | 173956 | 32772 | 5.29 | 0.0% | 95.7% | 0.5% | 3.4% | 0.4% | 37.6% | 4.98 |
| E | 177989 | 6640 | 26.62 | 0.0% | 0.0% | 37.0% | 61.2% | 1.7% | 57.0% | 4.91 |
Same diagnostics from the OFFLINE fixture replay (`nim c -r --path:common_libs common_libs/tests/test_tfil_commit_env.nim`, 20026 ticks, no Java):
| arm | interval | reversal rate |
|---|---:|---:|
| A control | 5.07 | 33.7% |
| B commit | 14.65 | 41.1% |
| C commit+noRev | 14.65 | 32.9% |
| D enemyTile | 4.94 | 35.6% |
| E commit30 | 27.57 | 42.0% |
The offline and live diagnostics agree on every arm: the interval moves off ~5 calls exactly where the arm says it should, so every treatment applied.
### Mean self-speed MAGNITUDE abs(sp) by offset (in calls) around a REVERSAL pick
This is the series that tests the premise 'the reversal collapses the bot's speed'. `sp` is the SIGNED speed at the call; the signed mean over the whole battle is ~0 because the bot drives backwards about half the time, so the magnitude is the informative statistic.
| arm | -5 | -4 | -3 | -2 | -1 | +0 | +1 | +2 | +3 | +4 | +5 | +6 | +7 | +8 | +9 | +10 |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| A | 4.61 | 4.61 | 4.64 | 4.69 | 4.77 | 4.84 | 4.19 | 3.95 | 4.05 | 4.19 | 4.32 | 4.45 | 4.54 | 4.65 | 4.75 | 4.83 |
| B | 5.96 | 5.95 | 5.85 | 5.73 | 5.55 | 5.28 | 4.32 | 3.91 | 4.10 | 4.49 | 4.90 | 5.24 | 5.53 | 5.76 | 5.92 | 6.01 |
| C | 6.07 | 6.04 | 5.97 | 5.84 | 5.66 | 5.32 | 4.38 | 3.95 | 4.09 | 4.46 | 4.81 | 5.21 | 5.59 | 5.94 | 6.11 | 6.21 |
| D | 5.07 | 5.02 | 5.00 | 5.02 | 5.00 | 4.94 | 4.38 | 4.21 | 4.33 | 4.50 | 4.64 | 4.75 | 4.85 | 4.98 | 5.06 | 5.11 |
| E | 5.00 | 4.93 | 4.84 | 4.78 | 4.67 | 4.52 | 4.19 | 4.08 | 4.22 | 4.42 | 4.58 | 4.77 | 4.98 | 5.18 | 5.32 | 5.40 |
(reversal samples per arm: A=12250, B=6125, C=5167, D=12336, E=3781)
## 5. Direct answer
- control (shipped mover): damage/run 274.44, round wins/run 2.79 (39/98), hits taken/run 96.50, reversal-pick rate 33.9%, mean abs(speed) 4.69
- arm B (B honour commitment): damage/run 274.07 (delta -0.37, p=0.9720) -> not distinguishable from control; round wins/run 3.07 (delta 0.29, p=0.6561) -> not distinguishable from control; hits taken/run 89.00 (delta -7.50, p=0.0577); reversal-pick rate 50.9% (control 33.9%)
- arm C (C B + no-reversal): damage/run 257.34 (delta -17.10, p=0.1559) -> not distinguishable from control; round wins/run 2.43 (delta -0.36, p=0.4720) -> not distinguishable from control; hits taken/run 87.21 (delta -9.29, p=0.0022); reversal-pick rate 42.4% (control 33.9%)
- arm D (D enemy-keyed cancel): damage/run 286.61 (delta 12.17, p=0.3567) -> not distinguishable from control; round wins/run 3.29 (delta 0.50, p=0.3500) -> not distinguishable from control; hits taken/run 92.50 (delta -4.00, p=0.2132); reversal-pick rate 37.6% (control 33.9%)
- arm E (E commit 30 ticks): damage/run 265.29 (delta -9.15, p=0.5574) -> not distinguishable from control; round wins/run 2.93 (delta 0.14, p=0.8865) -> not distinguishable from control; hits taken/run 92.36 (delta -4.14, p=0.2078); reversal-pick rate 57.0% (control 33.9%)
**NO ARM improves damage/run or round wins vs the shipped mover at p < 0.05.** Best point estimate on damage/run is arm D (delta 12.17); best on round wins/run is arm D (delta 0.50). Neither is statistically distinguishable from control at this n.
PREMISE CHECK (the bug report claimed the tile-change replan cancels the commitment, the reversals it causes are what get us hit):
- The cancel IS real and dominant in the shipped mover: arm A ends its commitment because of OUR tile change for 96.6% of picks at a 5.05-call interval. Arms B/C/E remove it (0.0% tile_self replans, interval 14.77 / 26.62) and arm D re-keys it to the enemy tile (95.7% tile_enemy).
- But honouring the commitment does NOT reduce reversal picks: the reversal rate RISES from 33.9% (control) to 50.9% (B) and 57.0% (E). The no-reversal arm C only claws part of that back (42.4%), still above control.
- The speed dip after a reversal is present in EVERY arm (see the profile table: ~4.9 -> ~4.0 px/tick at offsets +1..+2), i.e. it is a property of turning around, not of the tile replan. Arm A -- the arm carrying the bug -- actually has the LOWEST mean abs(speed).
common_libs/tests/fixtures/tfil_commit_ab_report_runs7.md
Vendored
+222
View File
@@ -0,0 +1,222 @@
# TFIL commitment A/B — measured
PRE-REGISTERED PRIMARY BLOCK: runs 1-7 per arm (35 battles, 245 rounds), 5 arms in parallel, 7 runs x 7 rounds each.
source commit: `19bf461`
frozen ModularBot binary sha256: `46e7ce19cc7889dacc0c5add9b29928dadb757fbc6ecbdb7765cddd7813361c0`
raw artifacts (NOT committed, live only): `/tmp/tfil_ab2/out/<arm>/run<n>.jsonl` + `...events.jsonl` and `/tmp/tfil_ab2/logs/<arm>/run<n>.out` + `...commit.jsonl`
analysed runs per arm: 7 (this file is self-consistent; re-run the tool with --runs N on the raw dir for a different block)
damage = server BulletHitBotEvent damage, attributed by numeric owner/victim id and verified against the capture's own counters; round wins = final RESULTS `firstPlaces`, cross-checked against BotDeathEvent.
## 1. Arm table (7 runs x 7 rounds vs real DrussGT, binary frozen from one source commit)
| arm | env | damage/run | damage taken/run | ROUND WINS | round win % | our shots/run | hits taken/run | our score/run | ticks/run |
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|
| A control (shipped) | `` | 280.38 | 192.41 | 19/49 | 38.8% | 830.7 | 95.7 | 460.7 | 1883 |
| B honour commitment | `TR_TFIL_TILE_REPLAN=off` | 281.05 | 194.69 | 23/49 | 46.9% | 794.1 | 90.0 | 503.3 | 1804 |
| C B + no-reversal | `TR_TFIL_TILE_REPLAN=off TR_TFIL_NO_REV=1` | 250.51 | 196.93 | 17/49 | 34.7% | 809.4 | 85.7 | 414.3 | 1871 |
| D enemy-keyed cancel | `TR_TFIL_TILE_REPLAN=enemy` | 301.36 | 211.31 | 24/49 | 49.0% | 791.4 | 93.3 | 521.7 | 1816 |
| E commit 30 ticks | `TR_TFIL_TILE_REPLAN=off TR_TFIL_COMMIT_TICKS=30` | 285.89 | 196.51 | 24/49 | 49.0% | 793.0 | 96.1 | 508.0 | 1822 |
## 2. Per-run values
**A control (shipped)** ``
| run | damage/run | damage taken | wins | round-by-round | hits taken | our shots | our score | enemy score | ticks |
|---:|---:|---:|---:|---|---:|---:|---:|---:|---:|
| 1 | 245.37 | 218.70 | 2/7 | L W L L L W L | 96 | 705 | 387 | 549 | 11166 |
| 2 | 256.58 | 240.50 | 1/7 | W L L L L L L | 101 | 828 | 324 | 640 | 13516 |
| 3 | 283.54 | 197.00 | 2/7 | L L L L W L W | 86 | 788 | 423 | 524 | 12607 |
| 4 | 306.11 | 198.00 | 3/7 | L L W L W W L | 95 | 848 | 509 | 461 | 13146 |
| 5 | 267.51 | 147.60 | 3/7 | W L L W L W L | 92 | 849 | 477 | 404 | 13483 |
| 6 | 271.07 | 138.80 | 4/7 | L W W L W W L | 88 | 885 | 546 | 327 | 13895 |
| 7 | 332.48 | 206.30 | 4/7 | W W L L W L W | 112 | 912 | 559 | 418 | 14461 |
**B honour commitment** `TR_TFIL_TILE_REPLAN=off`
| run | damage/run | damage taken | wins | round-by-round | hits taken | our shots | our score | enemy score | ticks |
|---:|---:|---:|---:|---|---:|---:|---:|---:|---:|
| 1 | 270.09 | 155.90 | 2/7 | L W L W L L L | 90 | 836 | 410 | 480 | 13267 |
| 2 | 263.83 | 234.60 | 1/7 | L L L L W L L | 111 | 853 | 333 | 631 | 13861 |
| 3 | 280.10 | 189.00 | 5/7 | W L W L W W W | 71 | 721 | 624 | 321 | 11326 |
| 4 | 307.40 | 139.70 | 6/7 | W W W W W W L | 75 | 821 | 707 | 206 | 13196 |
| 5 | 249.24 | 266.20 | 2/7 | L L W L L L W | 99 | 738 | 388 | 594 | 11626 |
| 6 | 275.90 | 161.30 | 4/7 | W L W W L L W | 87 | 812 | 550 | 360 | 12712 |
| 7 | 320.81 | 216.10 | 3/7 | W L L W W L L | 97 | 778 | 511 | 477 | 12428 |
**C B + no-reversal** `TR_TFIL_TILE_REPLAN=off TR_TFIL_NO_REV=1`
| run | damage/run | damage taken | wins | round-by-round | hits taken | our shots | our score | enemy score | ticks |
|---:|---:|---:|---:|---|---:|---:|---:|---:|---:|
| 1 | 240.31 | 200.90 | 3/7 | W L W L L W L | 89 | 787 | 446 | 467 | 13019 |
| 2 | 298.51 | 138.00 | 3/7 | W L L L W L W | 86 | 830 | 500 | 398 | 13598 |
| 3 | 244.43 | 210.30 | 2/7 | W L L W L L L | 74 | 803 | 379 | 545 | 12935 |
| 4 | 235.60 | 244.40 | 3/7 | L W W L W L L | 80 | 789 | 434 | 501 | 12292 |
| 5 | 268.72 | 193.20 | 3/7 | W L W W L L L | 93 | 839 | 466 | 458 | 13448 |
| 6 | 263.44 | 177.00 | 3/7 | W L L L L W W | 82 | 828 | 470 | 440 | 13385 |
| 7 | 202.55 | 214.70 | 0/7 | L L L L L L L | 96 | 790 | 205 | 676 | 13037 |
**D enemy-keyed cancel** `TR_TFIL_TILE_REPLAN=enemy`
| run | damage/run | damage taken | wins | round-by-round | hits taken | our shots | our score | enemy score | ticks |
|---:|---:|---:|---:|---|---:|---:|---:|---:|---:|
| 1 | 247.43 | 219.90 | 2/7 | W L W L L L L | 94 | 774 | 386 | 544 | 12775 |
| 2 | 332.70 | 202.50 | 5/7 | W W L W W L W | 104 | 864 | 624 | 350 | 13863 |
| 3 | 264.08 | 201.30 | 4/7 | L W L W L W W | 78 | 765 | 488 | 422 | 12557 |
| 4 | 276.78 | 215.00 | 1/7 | L L L L L W L | 97 | 806 | 332 | 612 | 12857 |
| 5 | 339.18 | 259.40 | 3/7 | W W L W L L L | 104 | 770 | 552 | 536 | 12453 |
| 6 | 321.08 | 161.70 | 4/7 | L L W W W L W | 89 | 809 | 601 | 351 | 12585 |
| 7 | 328.30 | 219.40 | 5/7 | W L W W W L W | 87 | 752 | 669 | 360 | 11942 |
**E commit 30 ticks** `TR_TFIL_TILE_REPLAN=off TR_TFIL_COMMIT_TICKS=30`
| run | damage/run | damage taken | wins | round-by-round | hits taken | our shots | our score | enemy score | ticks |
|---:|---:|---:|---:|---|---:|---:|---:|---:|---:|
| 1 | 316.56 | 215.20 | 3/7 | W W L L W L L | 110 | 847 | 513 | 483 | 13347 |
| 2 | 316.14 | 206.00 | 3/7 | W W L W L L L | 101 | 776 | 520 | 468 | 12588 |
| 3 | 271.46 | 206.20 | 3/7 | L L W L W W L | 104 | 788 | 441 | 498 | 12580 |
| 4 | 300.07 | 195.70 | 5/7 | L W L W W W W | 87 | 754 | 596 | 352 | 12233 |
| 5 | 256.01 | 188.80 | 3/7 | W L L W W L L | 94 | 828 | 465 | 452 | 13629 |
| 6 | 277.83 | 190.80 | 3/7 | W W L W L L L | 90 | 749 | 485 | 457 | 12302 |
| 7 | 263.14 | 172.90 | 4/7 | W L W W W L L | 87 | 809 | 536 | 374 | 12614 |
### Data integrity: id mapping + round-win cross-check
Round wins are taken from the final RESULTS `firstPlaces` and cross-checked against the BotDeathEvent of each round (in a TR 1v1 the round loser is the bot that died). A `WARNING` here means the two disagreed, i.e. do not trust that run.
- A/run1: id 1==ModularBot, id 2==DrussGT; of 1536 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.9% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- A/run2: id 1==ModularBot, id 2==DrussGT; of 1855 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 44.6% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- A/run3: id 1==ModularBot, id 2==DrussGT; of 1741 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.3% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- A/run4: id 2==ModularBot, id 1==DrussGT; of 1806 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 47.0% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- A/run5: id 1==ModularBot, id 2==DrussGT; of 1878 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.2% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- A/run6: id 2==ModularBot, id 1==DrussGT; of 1947 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.5% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- A/run7: id 2==ModularBot, id 1==DrussGT; of 1984 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 46.0% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- B/run1: id 2==ModularBot, id 1==DrussGT; of 1829 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.7% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- B/run2: id 1==ModularBot, id 2==DrussGT; of 1893 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.1% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- B/run3: id 2==ModularBot, id 1==DrussGT; of 1574 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.8% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- B/run4: id 2==ModularBot, id 1==DrussGT; of 1787 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.9% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- B/run5: id 1==ModularBot, id 2==DrussGT; of 1606 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 46.0% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- B/run6: id 2==ModularBot, id 1==DrussGT; of 1759 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 46.2% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- B/run7: id 2==ModularBot, id 1==DrussGT; of 1679 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 46.3% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- C/run1: id 1==ModularBot, id 2==DrussGT; of 1768 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 44.5% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- C/run2: id 1==ModularBot, id 2==DrussGT; of 1869 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 44.4% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- C/run3: id 2==ModularBot, id 1==DrussGT; of 1778 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.2% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- C/run4: id 1==ModularBot, id 2==DrussGT; of 1671 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 47.2% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- C/run5: id 2==ModularBot, id 1==DrussGT; of 1831 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.8% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- C/run6: id 1==ModularBot, id 2==DrussGT; of 1823 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.4% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- C/run7: id 1==ModularBot, id 2==DrussGT; of 1776 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 44.5% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- D/run1: id 1==ModularBot, id 2==DrussGT; of 1739 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 44.5% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- D/run2: id 2==ModularBot, id 1==DrussGT; of 1868 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 46.3% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- D/run3: id 1==ModularBot, id 2==DrussGT; of 1680 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.5% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- D/run4: id 1==ModularBot, id 2==DrussGT; of 1770 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.5% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- D/run5: id 1==ModularBot, id 2==DrussGT; of 1695 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.4% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- D/run6: id 2==ModularBot, id 1==DrussGT; of 1759 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 46.0% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- D/run7: id 2==ModularBot, id 1==DrussGT; of 1655 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.4% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- E/run1: id 2==ModularBot, id 1==DrussGT; of 1839 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 46.1% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- E/run2: id 2==ModularBot, id 1==DrussGT; of 1731 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 44.8% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- E/run3: id 2==ModularBot, id 1==DrussGT; of 1727 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.6% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- E/run4: id 2==ModularBot, id 1==DrussGT; of 1647 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.8% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- E/run5: id 2==ModularBot, id 1==DrussGT; of 1848 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 44.8% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- E/run6: id 2==ModularBot, id 1==DrussGT; of 1657 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 45.2% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
- E/run7: id 2==ModularBot, id 1==DrussGT; of 1760 fire events the bullet origin is within 60px of the firer's own recorded position for 100.0%, and 46.0% vote the same side (a 50% split would mean the mapping is noise) | WARNING: id mapping is unreliable
## 3. Exact two-sided permutation tests vs arm A (control = shipped mover)
Per-run test: all C(14,7)=3432 relabellings of the 7+7 per-run values; statistic = difference of means.
| metric | A mean | arm | arm mean | delta (arm - A) | exact two-sided p | perms |
|---|---:|---|---:|---:|---:|---:|
| damage/run | 280.38 | B | 281.05 | 0.67 | 0.9650 | 3432 |
| damage/run | 280.38 | C | 250.51 | -29.87 | 0.0880 | 3432 |
| damage/run | 280.38 | D | 301.36 | 20.99 | 0.2617 | 3432 |
| damage/run | 280.38 | E | 285.89 | 5.51 | 0.7121 | 3432 |
| round wins/run | 2.71 | B | 3.29 | 0.57 | 0.6020 | 3432 |
| round wins/run | 2.71 | C | 2.43 | -0.29 | 0.8164 | 3432 |
| round wins/run | 2.71 | D | 3.43 | 0.71 | 0.4312 | 3432 |
| round wins/run | 2.71 | E | 3.43 | 0.71 | 0.3019 | 3432 |
| hits taken/run | 95.71 | B | 90.00 | -5.71 | 0.3904 | 3432 |
| hits taken/run | 95.71 | C | 85.71 | -10.00 | 0.0437 | 3432 |
| hits taken/run | 95.71 | D | 93.29 | -2.43 | 0.6463 | 3432 |
| hits taken/run | 95.71 | E | 96.14 | 0.43 | 0.9510 | 3432 |
| our shots/run | 830.71 | B | 794.14 | -36.57 | 0.2721 | 3432 |
| our shots/run | 830.71 | C | 809.43 | -21.29 | 0.4598 | 3432 |
| our shots/run | 830.71 | D | 791.43 | -39.29 | 0.2110 | 3432 |
| our shots/run | 830.71 | E | 793.00 | -37.71 | 0.2220 | 3432 |
| our score/run | 460.71 | B | 503.29 | 42.57 | 0.4936 | 3432 |
| our score/run | 460.71 | C | 414.29 | -46.43 | 0.3922 | 3432 |
| our score/run | 460.71 | D | 521.71 | 61.00 | 0.3019 | 3432 |
| our score/run | 460.71 | E | 508.00 | 47.29 | 0.2407 | 3432 |
| ticks/run | 13182.00 | B | 12630.86 | -551.14 | 0.3106 | 3432 |
| ticks/run | 13182.00 | C | 13102.00 | -80.00 | 0.8660 | 3432 |
| ticks/run | 13182.00 | D | 12718.86 | -463.14 | 0.3362 | 3432 |
| ticks/run | 13182.00 | E | 12756.14 | -425.86 | 0.3730 | 3432 |
### Round-level test (exact, anti-conservative)
Sum statistic over all rounds of the arm vs arm A only; exact hypergeometric null. ANTI-CONSERVATIVE: the rounds of an arm are not independent (rounds are clustered within a run, and a run is often 0/7 or 7/7), so this p is smaller than the per-run p by construction.
| arm | rounds won | rounds lost | all rounds pooled | exact p (round level, vs A) |
|---|---:|---:|---|---:|
| A | 19 | 30 | 19/49 | 1.0000 |
| B | 23 | 26 | 23/49 | 0.5406 |
| C | 17 | 32 | 17/49 | 0.8342 |
| D | 24 | 25 | 24/49 | 0.4157 |
| E | 24 | 25 | 24/49 | 0.4157 |
## 4. TREATMENT diagnostics (TR_TFIL_COMMIT_LOG — did the arm actually apply?)
| arm | commit calls | picks | interval (calls) | % tile_self replans | % tile_enemy | % danger | % expiry | % init | reversal-pick rate | mean abs(speed) |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| A | 91795 | 18185 | 5.04 | 96.7% | 0.0% | 0.5% | 2.5% | 0.3% | 34.1% | 4.69 |
| B | 87863 | 5920 | 14.78 | 0.0% | 0.0% | 4.5% | 94.6% | 0.9% | 51.5% | 5.37 |
| C | 91331 | 6142 | 14.80 | 0.0% | 0.0% | 4.4% | 94.7% | 0.9% | 42.4% | 5.45 |
| D | 88392 | 16595 | 5.31 | 0.0% | 95.8% | 0.5% | 3.3% | 0.3% | 37.4% | 4.99 |
| E | 88864 | 3327 | 26.52 | 0.0% | 0.0% | 38.5% | 59.8% | 1.7% | 57.4% | 4.92 |
Same diagnostics from the OFFLINE fixture replay (`nim c -r --path:common_libs common_libs/tests/test_tfil_commit_env.nim`, 20026 ticks, no Java):
| arm | interval | reversal rate |
|---|---:|---:|
| A control | 5.07 | 33.7% |
| B commit | 14.65 | 41.1% |
| C commit+noRev | 14.65 | 32.9% |
| D enemyTile | 4.94 | 35.6% |
| E commit30 | 27.57 | 42.0% |
The offline and live diagnostics agree on every arm: the interval moves off ~5 calls exactly where the arm says it should, so every treatment applied.
### Mean self-speed MAGNITUDE abs(sp) by offset (in calls) around a REVERSAL pick
This is the series that tests the premise 'the reversal collapses the bot's speed'. `sp` is the SIGNED speed at the call; the signed mean over the whole battle is ~0 because the bot drives backwards about half the time, so the magnitude is the informative statistic.
| arm | -5 | -4 | -3 | -2 | -1 | +0 | +1 | +2 | +3 | +4 | +5 | +6 | +7 | +8 | +9 | +10 |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| A | 4.67 | 4.67 | 4.71 | 4.75 | 4.84 | 4.89 | 4.24 | 4.00 | 4.10 | 4.25 | 4.39 | 4.52 | 4.59 | 4.68 | 4.79 | 4.87 |
| B | 5.95 | 5.93 | 5.87 | 5.76 | 5.58 | 5.37 | 4.38 | 3.95 | 4.12 | 4.53 | 4.96 | 5.29 | 5.60 | 5.78 | 5.87 | 5.94 |
| C | 6.08 | 6.02 | 5.94 | 5.83 | 5.66 | 5.31 | 4.46 | 4.06 | 4.21 | 4.55 | 4.87 | 5.23 | 5.58 | 5.90 | 6.04 | 6.08 |
| D | 5.15 | 5.10 | 5.07 | 5.06 | 5.03 | 4.98 | 4.42 | 4.27 | 4.39 | 4.56 | 4.70 | 4.82 | 4.90 | 5.04 | 5.12 | 5.15 |
| E | 5.03 | 4.97 | 4.88 | 4.81 | 4.66 | 4.54 | 4.26 | 4.17 | 4.28 | 4.44 | 4.58 | 4.76 | 4.94 | 5.12 | 5.27 | 5.35 |
(reversal samples per arm: A=6200, B=3047, C=2602, D=6199, E=1910)
## 5. Direct answer
- control (shipped mover): damage/run 280.38, round wins/run 2.71 (19/49), hits taken/run 95.71, reversal-pick rate 34.1%, mean abs(speed) 4.69
- arm B (B honour commitment): damage/run 281.05 (delta 0.67, p=0.9650) -> not distinguishable from control; round wins/run 3.29 (delta 0.57, p=0.6020) -> not distinguishable from control; hits taken/run 90.00 (delta -5.71, p=0.3904); reversal-pick rate 51.5% (control 34.1%)
- arm C (C B + no-reversal): damage/run 250.51 (delta -29.87, p=0.0880) -> not distinguishable from control; round wins/run 2.43 (delta -0.29, p=0.8164) -> not distinguishable from control; hits taken/run 85.71 (delta -10.00, p=0.0437); reversal-pick rate 42.4% (control 34.1%)
- arm D (D enemy-keyed cancel): damage/run 301.36 (delta 20.99, p=0.2617) -> not distinguishable from control; round wins/run 3.43 (delta 0.71, p=0.4312) -> not distinguishable from control; hits taken/run 93.29 (delta -2.43, p=0.6463); reversal-pick rate 37.4% (control 34.1%)
- arm E (E commit 30 ticks): damage/run 285.89 (delta 5.51, p=0.7121) -> not distinguishable from control; round wins/run 3.43 (delta 0.71, p=0.3019) -> not distinguishable from control; hits taken/run 96.14 (delta 0.43, p=0.9510); reversal-pick rate 57.4% (control 34.1%)
**NO ARM improves damage/run or round wins vs the shipped mover at p < 0.05.** Best point estimate on damage/run is arm D (delta 20.99); best on round wins/run is arm D (delta 0.71). Neither is statistically distinguishable from control at this n.
PREMISE CHECK (the bug report claimed the tile-change replan cancels the commitment, the reversals it causes are what get us hit):
- The cancel IS real and dominant in the shipped mover: arm A ends its commitment because of OUR tile change for 96.7% of picks at a 5.04-call interval. Arms B/C/E remove it (0.0% tile_self replans, interval 14.78 / 26.52) and arm D re-keys it to the enemy tile (95.8% tile_enemy).
- But honouring the commitment does NOT reduce reversal picks: the reversal rate RISES from 34.1% (control) to 51.5% (B) and 57.4% (E). The no-reversal arm C only claws part of that back (42.4%), still above control.
- The speed dip after a reversal is present in EVERY arm (see the profile table: ~4.9 -> ~4.0 px/tick at offsets +1..+2), i.e. it is a property of turning around, not of the tile replan. Arm A -- the arm carrying the bug -- actually has the LOWEST mean abs(speed).
common_libs/tests/fixtures/tfil_commit_ab_results_runs14.json
Vendored
+2936
View File
File diff suppressed because it is too large Load Diff
common_libs/tests/fixtures/tfil_commit_ab_results_runs7.json
Vendored
+1662
View File
File diff suppressed because it is too large Load Diff