Added the four automata-level metrics to the TM diagnostics kit (settledness,
clause diversity, churn, vote disagreement) plus a state histogram, a per-input
confidence table and a one-line health summary, and validated them on a
learnable-vs-noise pair.
STATE CONVENTIONS, read off OUR code rather than from memory:
range [-nStates, nStates] as int16; nStates = 64 for tm_pattern, 32 for tsetlin
initial value 0 = the Exclude boundary
INCLUDE iff state > 0; EXCLUDE iff state <= 0
flip boundary sits between state 0 and 1; commitment = abs(st)/nStates in [0,1]
=== THE GATE, AND A RESULT THAT MATTERS ===
Case A (learnable planted rule) vs Case B (shuffled labels), 49 bits, N=64:
metric A (learnable) B (shuffled)
settledness mean 0.970 0.719
churn flip/sample 0.000055 FALLING 0.000788 FLAT
clause-change/sample 0.00263 falling 0.0595 flat
diversity (Jaccard) 0.176 0.014
disagreement 0.003 0.298
verdict settling mixed (NOT settling)
**SETTLEDNESS ALONE DOES NOT WORK.** On noise the automata still COMMIT (0.719) -
they just commit to the wrong thing. The decisive separators are **churn TREND
(falling vs flat)** and **vote DISAGREEMENT (0.003 vs 0.298)**. Had we built only
the settledness metric - the one that seems most obvious - we would have been
misled. That is now recorded in the README.
INERTIA SWEEP: A vs B separate at N=16/32/64/128. **Raising N raises A's
commitment but does NOT reduce B's noise-fitting** - so more inertia does not
rescue a noise-fitting TM.
=== REAL READING ON THE SHIPPED GUN, AND THE INFERENCE IT SUPPORTS ===
tm_pattern GF head over the DrussGT fixtures: settledness 0.484 (settling),
diversity 0.267 (moderate), churn 0.094/100 FALLING, disagreement 0.145
(coherent). **VERDICT: SETTLING** - not fidgeting, not collapsed. Constant inputs
flagged: 38/39 (the known never-written bits) plus 19/36/37.
Context: pooled warm accuracy 35.72% vs 34.24% majority = +1.48pp.
So: **the old gun was NOT failing because of inertia or instability - it settled
properly and its settled rules still barely beat a lazy guess.** Its settledness
(0.484) is LOWER than both synthetic cases (0.97/0.72), which is the signature of
WEAK OR CONFLICTING SIGNAL rather than too much inertia.
CONCLUSION: **N and s are not the observed bottleneck. The target/representation
is.** That is exactly why the new design changes the target and the label
pipeline rather than sweeping knobs - and it means we should NOT spend effort on
an N/s sweep expecting it to fix anything.
Also adds `diag_automata_validation.nim` (Case A/B/C + inertia sweep) and
`test_tm_automata_diag.nim` (55 pure checks); `test_tm_diag` 48 and
`diag_synthetic` 17 still pass, plus all other guards. acceptance_offline_vs_online
was NOT run (it needs a live battle and there is no tm_diag dependency).
Caveat: churn on the real gun is a PROXY (a tm_core retrain over captured samples
in live order) because the live gun exposes no per-sample state trace; the other
metrics are read directly off the exported teams.
SirStone
ab8d383121