# Gun Predictor Shootout — Final Report

## Executive Summary

WiSARD K=14 (bleach=1, 276-bit input) wins the hyperparameter sweep with MAE=1.37px on the single-enemy multi-frame dataset. In head-to-head battle backtest across 4 enemy types, WiSARD K=12 beats both BNNBot (linear+Hebbian) and TsetlinBot on 3/4 enemies, with an overall MAE of 16.20 vs 17.61 (BNNBot) and 20.69 (TsetlinBot). The Tsetlin Machine is the strongest warm-phase predictor in isolation (MAE(last⅓)=0.27 for best config) but its cold-start overhead costs it in battles with <~300 ticks of data. Ship WiSARD K=14, bleach=1. If cold-start penalty on TM is ever fixed (e.g., pre-training or warm-up from WiSARD), TM becomes worth revisiting.

---

## Methods Tested

**16 method families, ~260 configurations total:**

- WiSARD (192 configs: K=4–20, bleach=0–3, ±XOR features)
- Tsetlin Machine / RTM (57 configs: 10–500 clauses, s=1.5–15, T=10–200, ±weighted)
- Echo State Network (reservoir=512)
- Kanerva Sparse Distributed Memory (addr=2000)
- N-gram Markov predictor (4×69 chunks)
- Bloom Filter (8192 slots)
- Hyperdimensional Computing / VSA (n_hd=2000, 32 classes)
- Random Subspace ensemble (30×50 bits)
- WiSARD + Eligibility Traces (k=12, trace=5)

---

## Results Table

> **Dataset note:** WiSARD sweep and alternatives used a 277–3153 row single-enemy dataset at rep power p1.07. Battle comparison used 73 380 data points across 4 enemy types and all bullet powers. MAE numbers are not directly comparable across the two columns — treat sweep MAE as a relative rank within each sweep, and battle MAE as the real-world number.

| Rank | Method | Best Config | MAE (sweep) | MAE (battle) | Hit% (battle) | Memory | Learning Speed | Verdict |
|------|--------|-------------|-------------|--------------|---------------|--------|----------------|---------|
| 1 | WiSARD | K=14, bleach=1, 276b | 1.37 px | 16.20 | 65.4% | ~276b active | fast | **SHIP IT** |
| 2 | WiSARD+Elig | k=12, trace=5 | 8.24* | — | 90.3%* | ~276b active | fast | promising, untested in battle |
| 3 | RandSubspace | 30×50 bits | 8.31* | — | 90.6%* | ~1500b | fast | similar to WiSARD, no battle test |
| 4 | Echo State Net | res=512 | 8.29* | — | 85.2%* | large (float reservoir) | med | too heavy for Robocode JVM |
| 5 | BNNBot (Lin+Heb) | linear+Hebbian, 24-cell | — | 17.61 | 61.9% | tiny | fast | current baseline |
| 6 | Tsetlin Machine | 500 cl, T=50, s=1.5, W=Y | 2.84† | 20.69 (warm: 18.77) | 55.8% | 531 KB | slow (cold-start) | future work |
| 7 | N-gram Markov | 4×69 chunks | 10.11* | — | 80.5%* | small | med | no improvement on WiSARD |
| 8 | Kanerva SDM | addr=2000 | 10.16* | — | 80.1%* | med | slow | no improvement on baseline |
| 9 | Bloom Filter | 8192 slots | 17.08* | — | 59.2%* | 8 KB | fast | worse than baseline |
| 10 | HDC/VSA | n_hd=2000, 32 cls | 65.33* | — | 0.0%* | large | slow | broken for this task |

*Measured on 277-row alternatives dataset (single enemy, p1.07).  
†Measured on 2500-row TM sweep dataset (single enemy).  
Battle column from 4-enemy head-to-head (73 380 rows, all powers).

---

## Per-Enemy Analysis

| Enemy | BNNBot MAE | WiSARD MAE | TsetlinBot MAE | Winner | Notes |
|-------|-----------|-----------|---------------|--------|-------|
| target | 2.68 | **2.21** | 4.14 | WiSARD | Predictable bot, all methods work; WiSARD widest margin |
| walls | 15.31 | **13.22** | 18.81 | WiSARD | Wall-bouncing pattern, WiSARD generalises better |
| crazy | 24.72 | **22.29** | 27.10 | WiSARD | Highly random; WiSARD still edges out |
| spinbot | **23.75** | 24.54 | 30.20 | BNNBot | Periodic rotation pattern; Hebbian table captures it |

**Specialist vs generalist:** WiSARD is the best generalist (3/4 wins). BNNBot's Hebbian residual table memorises spinbot's periodic pattern more effectively than WiSARD's RAM-based lookup. TM loses on all 4 — its cold start dominates the all-rows metric in battles of this length.

TM warm-phase on crazy (20.98) nearly matches WiSARD (22.29), suggesting TM catches up once trained but battles are typically too short for it to amortise the cold start.

---

## Key Insights

### WiSARD

- **K=14 is the sweet spot.** MAE plateaus K=12–20; diminishing returns above K=14. K=4–6 measurably worse.
- **Bleaching has marginal effect.** bleach=0 vs bleach=1 vs bleach=2 all within ±0.02 MAE. Skip tuning it; leave at 1.
- **XOR features add nothing.** The +XOR variants (345b input) are uniformly equal or worse than plain 276b for the same K. Extra bits, no gain.
- **Cold-start (MAE f50) still high** (~0.43–0.58 px for best configs), but last-50 converges to ~0.00 px — the model fully memorises the pattern after enough data.
- **Baseline delta:** best WiSARD is -0.26 px vs P1 linear, which sounds small but the absolute MAE of 1.37 vs 1.63 is a ~16% improvement at p1.07. Battle-scale improvement is larger (16.20 vs 17.61, ~8% reduction).

### Tsetlin Machine

- **Weighted clauses are mandatory.** Unweighted configs are consistently 0.5–2.0 MAE higher at identical clause counts.
- **s=1.5 is optimal.** Specificity 3.0 works, 5.0+ degrades, 10–15 is too sparse.
- **T (threshold) barely matters** across 10–200 range — vote clamping is rarely active on these dataset sizes.
- **Cold-start dominates all-rows MAE.** The "MAE(all)" column is misleading — it averages a bad warm-up phase over the whole run. The relevant metric for a real battle is MAE(last⅓): best config gets 0.27 px (!) for 500 clauses, 0.15 px for 100 clauses.
- **Practical config:** 100 clauses, T=100, s=1.5, states=32, weighted=Y. MAE(all)=3.28, MAE(last⅓)=0.20, 106 KB. The 500-clause config is marginally better but 5× the memory and slow.
- **Why it loses in battles:** RTM shared x/y with 60 clauses was used in the battle test — underspecified relative to the sweep's best config. Even so, TM warm-phase (18.77) still loses to WiSARD (16.20) overall.

### Alternatives

- **WiSARD+Eligibility Traces** is the real second-place finisher (MAE=8.24, Hit%=90.3% on the 277-row test). Eligibility traces let the RAM weights decay — this is strictly better than vanilla WiSARD on the small-dataset test. Not yet battle-tested.
- **Random Subspace ensemble** (8.31, 90.6% hit) matches WiSARD+Elig and is simpler to implement, but also untested in the full battle setting.
- **ESN** (8.29) is competitive but requires a float reservoir — doesn't fit the binary/frugal constraint of this project.
- **N-gram and SDM** essentially replicate linear baseline performance. Not worth the complexity.
- **Bloom Filter** — worse than baseline. RAM addressed by pattern ID without content-addressing fails here.
- **HDC/VSA** — completely broken (0% hit rate). The 32-class bucketing is too coarse for continuous angular prediction; this architecture is wrong for regression.

### Eligibility Traces Finding

The WiSARD+Elig variant (trace=5) improves MAE by ~1.7 px and hit rate by +10 pp vs vanilla WiSARD K=12 on the same 277-row dataset. This is the largest single improvement found across all alternatives. Mechanism: eligibility traces propagate reinforcement to recently active RAM addresses, not just the current one — capturing temporal credit assignment that WiSARD's instantaneous lookup misses.

---

## Recommendation

**Ship:** WiSARD K=14, bleach=1, 276-bit input (4 frames × 69 bits), 23 RAM nodes.

```
K         = 14
bleach    = 1
input_bits = 276  # 4 frames × 69 bits, no XOR
```

This is the best-validated config with a real battle backtest (WiSARD K=12 wins 3/4 enemies; K=14 is marginally better in the sweep). Memory is ~276 bits active per RAM node, negligible for the JVM.

**Explore next (in priority order):**

1. **WiSARD+Eligibility Traces (k=14, trace=5)** — +10 pp hit rate in the alternatives test. Straightforward to implement on top of current WiSARD. Highest expected ROI.
2. **Tsetlin Machine with warm-start** — if you can pre-train TM on a stored trajectory from the prior round, cold-start vanishes and TM's MAE(last⅓)=0.20 becomes competitive. Requires inter-round state persistence.
3. **Random Subspace ensemble** — drop-in if eligibility traces are too complex; similar performance.
4. **Per-enemy specialisation** — spinbot is the one case where Hebbian residuals beat WiSARD. A simple enemy-classifier + model switcher could capture that.

Do not pursue: HDC, Bloom Filter, SDM. None improved on linear.

---

## Raw Data References

| File | Content |
|------|---------|
| `analysis/sweep_wisard_results.txt` | WiSARD sweep: 192 configs (K=4–20, bleach=0–3, ±XOR), 3153 rows, 3 seeds each |
| `analysis/sweep_tsetlin_results.txt` | TM sweep: 57 configs (clauses=10–500, s=1.5–15, T=10–200, ±weighted), 2500 rows |
| `analysis/alternatives_results.txt` | 7 alternative methods vs WiSARD K=12 baseline, 277 rows |
| `analysis/battle_comparison.txt` | Head-to-head battle backtest: BNNBot vs WiSARD K=12 vs TsetlinBot, 4 enemies, 73 380 data points |