2 Commits

Author	SHA1	Message	Date
SirStone	ab8d383121	Automata metrics: settledness alone does NOT separate learning from fidgeting ... Added the four automata-level metrics to the TM diagnostics kit (settledness, clause diversity, churn, vote disagreement) plus a state histogram, a per-input confidence table and a one-line health summary, and validated them on a learnable-vs-noise pair. STATE CONVENTIONS, read off OUR code rather than from memory: range [-nStates, nStates] as int16; nStates = 64 for tm_pattern, 32 for tsetlin initial value 0 = the Exclude boundary INCLUDE iff state > 0; EXCLUDE iff state <= 0 flip boundary sits between state 0 and 1; commitment = abs(st)/nStates in [0,1] === THE GATE, AND A RESULT THAT MATTERS === Case A (learnable planted rule) vs Case B (shuffled labels), 49 bits, N=64: metric A (learnable) B (shuffled) settledness mean 0.970 0.719 churn flip/sample 0.000055 FALLING 0.000788 FLAT clause-change/sample 0.00263 falling 0.0595 flat diversity (Jaccard) 0.176 0.014 disagreement 0.003 0.298 verdict settling mixed (NOT settling) **SETTLEDNESS ALONE DOES NOT WORK.** On noise the automata still COMMIT (0.719) - they just commit to the wrong thing. The decisive separators are **churn TREND (falling vs flat)** and **vote DISAGREEMENT (0.003 vs 0.298)**. Had we built only the settledness metric - the one that seems most obvious - we would have been misled. That is now recorded in the README. INERTIA SWEEP: A vs B separate at N=16/32/64/128. **Raising N raises A's commitment but does NOT reduce B's noise-fitting** - so more inertia does not rescue a noise-fitting TM. === REAL READING ON THE SHIPPED GUN, AND THE INFERENCE IT SUPPORTS === tm_pattern GF head over the DrussGT fixtures: settledness 0.484 (settling), diversity 0.267 (moderate), churn 0.094/100 FALLING, disagreement 0.145 (coherent). **VERDICT: SETTLING** - not fidgeting, not collapsed. Constant inputs flagged: 38/39 (the known never-written bits) plus 19/36/37. Context: pooled warm accuracy 35.72% vs 34.24% majority = +1.48pp. So: **the old gun was NOT failing because of inertia or instability - it settled properly and its settled rules still barely beat a lazy guess.** Its settledness (0.484) is LOWER than both synthetic cases (0.97/0.72), which is the signature of WEAK OR CONFLICTING SIGNAL rather than too much inertia. CONCLUSION: **N and s are not the observed bottleneck. The target/representation is.** That is exactly why the new design changes the target and the label pipeline rather than sweeping knobs - and it means we should NOT spend effort on an N/s sweep expecting it to fix anything. Also adds `diag_automata_validation.nim` (Case A/B/C + inertia sweep) and `test_tm_automata_diag.nim` (55 pure checks); `test_tm_diag` 48 and `diag_synthetic` 17 still pass, plus all other guards. acceptance_offline_vs_online was NOT run (it needs a live battle and there is no tm_diag dependency). Caveat: churn on the real gun is a PROXY (a tm_core retrain over captured samples in live order) because the live gun exposes no per-sample state trace; the other metrics are read directly off the exported teams.	2026-09-22 21:40:27 +02:00
SirStone	f9f8d84671	TM diagnostics kit: VALIDATED (finds a known dead input), and it found a real bug ... Built `common_libs/tm_diag/` as a first-class offline diagnostics kit for Tsetlin work, BEFORE writing the new gun - because we hit two data problems tonight that no amount of reading the TM's clauses would have revealed (a 38.8% majority answer, and 36-58% mislabelled training samples). WHAT IT PROVIDES - `feature_spec.nim`: a NAMED feature container, so a learned clause prints as a sentence (`IF near-wall AND bullet-dead-on AND turn-left(t-2) THEN class=3`) instead of "feature 17". Includes the 49-bit draft spec from the design session and the shipped 40-bit encoding. - `tm_core.nim`: a compact deterministic Granmo multiclass TM with an INTROSPECTABLE clause layout (mirrors the tm_pattern core). - `diagnostics.nim`, six groups: (1) pre-flight DATA checks + shuffled-label control, (2) clause introspection (readable dump, per-clause vote counts, empty and never-fired clauses, length distribution, per-class balance), (3) per-feature contribution with an explicit DEAD-INPUT LIST and a ranked most-valuable list, (4) accuracy vs the majority baseline with per-class precision/recall and pred-majority share, (5) learning curve, (6) ablation hooks (drop a block / scramble a bit). === TASK 3: THE VALIDATION THAT GATES EVERYTHING - PASSED WITH NUMBERS === A diagnostic we never checked is worthless, so the kit was tested on a synthetic set with a PLANTED RULE (class2 = A and B, class1 = A and not B, class0 = not A), a deliberately IRRELEVANT block (US, 9 bits) and a PURE-NOISE bit (17). - majority baseline 60.63% (class0); over-30% correctly flagged - **the planted rule is recovered EXACTLY** via `necessaryLiterals`: class0 IF NOT dist-wall<50 | class1 IF dist-wall<50 AND NOT lat DEAD-ON | class2 IF dist-wall<50 AND lat DEAD-ON - **DEAD-INPUT LIST = all 9 US bits AND the noise bit 17**, while the planted bits 0 and 45 are correctly NOT listed - top contributors: bit0 w=1241.7, bit45 w=583.3, then 49.8 - a 12-25x gap, so the relevant bits are unmistakable - **ABLATION: drop WALLS -39.47pp, drop BULLETS -19.33pp, drop US 0.00pp**, scramble A -42.00pp, scramble the noise bit 0.00pp - shuffled-label control 60.40% vs majority 60.63% = -0.23pp -> no leak So the kit reliably finds a known dead input and a known relevant one. === TASK 4: THE REAL READING, AND A BUG IN THE SHIPPED GUN === `tm_pattern` GF head, 6 DrussGT fixtures, pooled 250,745 samples: - label balance c2 = **34.4%** (majority-heavy, flagged); accuracy **35.72%** vs majority **34.24%** -> margin **+1.48pp**. On `tr_drussgt_vs_crazy` it is BELOW majority (33.81% vs 37.72%, -3.92pp). - 200 clauses: **27 empty, 45 never fired**, mean length 19.17, max 57. The majority class is starved (class2: 22 non-empty, 18 empty, only 2 positive fired). Class4 fires 11-24-literal clauses -> memorisation signature. - **REPRESENTATION BUG FOUND (reported, not silently fixed):** `tmBuildBits` writes only 38 raw bits into `var bits: array[TM_NBITS=40, uint8]` - bits 38 and 39 are NEVER ASSIGNED, so they are always 0 and their negated literals are always 1. The kit's `constantInputs` confirms 38/39 are constant, and **`UNUSED-38`/`39` rank #6 and #8 in the most-valuable-inputs list** - i.e. the model's highest-usage inputs are information-free. That is a representation bug, not a display artefact, and it is a concrete mechanism for part of the poor learning. DRAFT ENCODING CHECKED: the 49-bit draft is arithmetically consistent (4+4=8 walls, 6+3=9 us, 3+5+3+3+3+3=20 motion, 5+7=12 bullets = 49). No draft inconsistency. Guards: test_tm_diag 48 (new), diag_synthetic 17 (new), test_gun_harness 39, test_vbullet_metric 11, test_power_selection 3, test_adaptive_radar 41, test_tfil_ring_weights 24, test_power_policy 26, test_ram_decision 40, test_rack_membership 48, test_selector_tiebreak 19, test_tm_pattern_registration 20, test_vbullet_admit_gate 12, acceptance_offline_vs_online 12/12; tm_pattern_learning passes. The tm_pattern hook is additive and default-OFF (no behaviour change). NOT YET INCLUDED (the automata metrics discussed for the next step): per-clause automata settledness (distance from the flip point), clause diversity (pairwise overlap), literal-set churn over time, and cross-clause vote disagreement. The kit has clause-level diagnostics but not the automata-state ones.	2026-09-22 21:24:27 +02:00