Automata metrics: settledness alone does NOT separate learning from fidgeting
Added the four automata-level metrics to the TM diagnostics kit (settledness, clause diversity, churn, vote disagreement) plus a state histogram, a per-input confidence table and a one-line health summary, and validated them on a learnable-vs-noise pair. STATE CONVENTIONS, read off OUR code rather than from memory: range [-nStates, nStates] as int16; nStates = 64 for tm_pattern, 32 for tsetlin initial value 0 = the Exclude boundary INCLUDE iff state > 0; EXCLUDE iff state <= 0 flip boundary sits between state 0 and 1; commitment = abs(st)/nStates in [0,1] === THE GATE, AND A RESULT THAT MATTERS === Case A (learnable planted rule) vs Case B (shuffled labels), 49 bits, N=64: metric A (learnable) B (shuffled) settledness mean 0.970 0.719 churn flip/sample 0.000055 FALLING 0.000788 FLAT clause-change/sample 0.00263 falling 0.0595 flat diversity (Jaccard) 0.176 0.014 disagreement 0.003 0.298 verdict settling mixed (NOT settling) **SETTLEDNESS ALONE DOES NOT WORK.** On noise the automata still COMMIT (0.719) - they just commit to the wrong thing. The decisive separators are **churn TREND (falling vs flat)** and **vote DISAGREEMENT (0.003 vs 0.298)**. Had we built only the settledness metric - the one that seems most obvious - we would have been misled. That is now recorded in the README. INERTIA SWEEP: A vs B separate at N=16/32/64/128. **Raising N raises A's commitment but does NOT reduce B's noise-fitting** - so more inertia does not rescue a noise-fitting TM. === REAL READING ON THE SHIPPED GUN, AND THE INFERENCE IT SUPPORTS === tm_pattern GF head over the DrussGT fixtures: settledness 0.484 (settling), diversity 0.267 (moderate), churn 0.094/100 FALLING, disagreement 0.145 (coherent). **VERDICT: SETTLING** - not fidgeting, not collapsed. Constant inputs flagged: 38/39 (the known never-written bits) plus 19/36/37. Context: pooled warm accuracy 35.72% vs 34.24% majority = +1.48pp. So: **the old gun was NOT failing because of inertia or instability - it settled properly and its settled rules still barely beat a lazy guess.** Its settledness (0.484) is LOWER than both synthetic cases (0.97/0.72), which is the signature of WEAK OR CONFLICTING SIGNAL rather than too much inertia. CONCLUSION: **N and s are not the observed bottleneck. The target/representation is.** That is exactly why the new design changes the target and the label pipeline rather than sweeping knobs - and it means we should NOT spend effort on an N/s sweep expecting it to fix anything. Also adds `diag_automata_validation.nim` (Case A/B/C + inertia sweep) and `test_tm_automata_diag.nim` (55 pure checks); `test_tm_diag` 48 and `diag_synthetic` 17 still pass, plus all other guards. acceptance_offline_vs_online was NOT run (it needs a live battle and there is no tm_diag dependency). Caveat: churn on the real gun is a PROXY (a tm_core retrain over captured samples in live order) because the live gun exposes no per-sample state trace; the other metrics are read directly off the exported teams.
This commit is contained in:
@@ -1,20 +1,24 @@
|
||||
## tm_diag/diagnostics.nim — THE DIAGNOSTICS KIT (Task 2, six groups).
|
||||
## tm_diag/diagnostics.nim — THE DIAGNOSTICS KIT (Tasks 2 + 5).
|
||||
##
|
||||
## Everything runs OFFLINE: a trained `TmMachine` plus a labelled sample set.
|
||||
## No battles, no harness, no Java. The six groups are:
|
||||
## No battles, no harness, no Java. The seven groups are:
|
||||
## 1. pre-flight DATA checks (dataChecks, shuffledLabelControl)
|
||||
## 2. clause introspection (clauseInfo, clauseSummary, topClauses)
|
||||
## 3. per-feature contribution (featureContributions, deadInputs, ...)
|
||||
## 4. accuracy diagnostics (accuracyDiagnostics)
|
||||
## 5. learning curve (learningCurve)
|
||||
## 6. ablation hooks (ablateDropBlock, ablateScrambleFeature, ...)
|
||||
## 7. AUTOMATA level (Task 5) (settledness, clauseDiversity, churnTrace,
|
||||
## voteDisagreement, stateHistogram,
|
||||
## perInputConfidence, healthLine,
|
||||
## automataDiagnostics)
|
||||
##
|
||||
## A `DiagSample` carries the LITERAL vector (positive literals [0..nBits),
|
||||
## negations [nBits..2*nBits)) exactly as the shipped TM stores it, plus the
|
||||
## true label. This keeps the kit usable both on the reference core and on the
|
||||
## literal vectors captured from the real gun.
|
||||
|
||||
import std/[math, strformat, algorithm]
|
||||
import std/[math, strformat, algorithm, strutils]
|
||||
import feature_spec
|
||||
import tm_core
|
||||
|
||||
@@ -587,3 +591,551 @@ proc ablateBaseline*(tmpl: TmMachine, train, eval: openArray[DiagSample],
|
||||
baselineAcc: evalAcc(trainModel(tmpl, train, epochs, seed), eval),
|
||||
ablatedAcc: evalAcc(trainModel(tmpl, train, epochs, seed), eval),
|
||||
delta: 0.0)
|
||||
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
# GROUP 7 — automata-level metrics
|
||||
# settledness / clause diversity / churn / vote disagreement
|
||||
# plus the state histogram, the per-input confidence table and a health line.
|
||||
#
|
||||
# STATE CONVENTIONS — derived from the ACTUAL code (tm_core.nim and
|
||||
# guns/tm_pattern.nim), not from the textbook formulation:
|
||||
# * every clause owns one automaton per literal (2 * nBits of them);
|
||||
# * states are int16 in [-nStates, nStates];
|
||||
# * INCLUDE iff state > 0, EXCLUDE iff state <= 0 (tmEval / clauseLits);
|
||||
# * both cores initialise every state to 0 ("the Exclude boundary"), so the
|
||||
# flip boundary sits BETWEEN state 0 (Exclude) and state 1 (Include) — the
|
||||
# middle of the range.
|
||||
# * commitment(st) = |st| / nStates in [0, 1]: 0 exactly on the boundary,
|
||||
# 1 at either extreme. It is symmetric for the two sides.
|
||||
# ─────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
proc stateCommitment*(nStates, st: int): float {.inline.} =
|
||||
## Normalised distance of an automaton state from its flip boundary, in
|
||||
## [0, 1]. 0 == on the boundary (state 0), 1 == at either extreme (+/-nStates).
|
||||
min(abs(st).float / max(1, nStates).float, 1.0)
|
||||
|
||||
proc stateIncluded*(st: int): bool {.inline.} =
|
||||
## The kit's include predicate, matching `tmEval` / `clauseLits` exactly.
|
||||
st > 0
|
||||
|
||||
# ── 1. SETTLEDNESS ───────────────────────────────────────────────────────────
|
||||
|
||||
type
|
||||
ClauseSettledness* = object
|
||||
cls*: int
|
||||
index*: int
|
||||
polarity*: int ## +1 positive clauses, -1 negative clauses
|
||||
nAutomata*: int
|
||||
meanCommitment*: float
|
||||
settledFraction*: float
|
||||
|
||||
SettlednessResult* = object
|
||||
threshold*: float
|
||||
nAutomata*: int
|
||||
overallMean*: float
|
||||
overallSettledFraction*: float
|
||||
clauses*: seq[ClauseSettledness]
|
||||
|
||||
proc settledness*(m: TmMachine, threshold = 0.5): SettlednessResult =
|
||||
## Per clause and overall: the mean automaton `commitment` and the fraction of
|
||||
## automata whose commitment is >= `threshold`. A clause full of automata
|
||||
## parked near the boundary is wavering noise; one with automata pushed to an
|
||||
## extreme has committed to a rule.
|
||||
result.threshold = threshold
|
||||
var sum = 0.0
|
||||
var settled = 0
|
||||
for c in 0..<m.nClasses:
|
||||
for cl in 0..<m.nClauses:
|
||||
let base = cl * m.nLiterals
|
||||
var csum = 0.0
|
||||
var csettled = 0
|
||||
for lit in 0..<m.nLiterals:
|
||||
let cm = stateCommitment(m.nStates, int(m.teams[c][base + lit]))
|
||||
csum += cm
|
||||
if cm >= threshold: inc csettled
|
||||
result.clauses.add ClauseSettledness(
|
||||
cls: c, index: cl,
|
||||
polarity: (if cl < m.half: 1 else: -1),
|
||||
nAutomata: m.nLiterals,
|
||||
meanCommitment: csum / m.nLiterals.float,
|
||||
settledFraction: csettled.float / m.nLiterals.float)
|
||||
sum += csum
|
||||
settled += csettled
|
||||
inc result.nAutomata, m.nLiterals
|
||||
result.overallMean =
|
||||
if result.nAutomata > 0: sum / result.nAutomata.float else: 0.0
|
||||
result.overallSettledFraction =
|
||||
if result.nAutomata > 0: settled.float / result.nAutomata.float else: 0.0
|
||||
|
||||
proc settlednessTrend*(early, late: SettlednessResult): string =
|
||||
## Classify whether settledness ROSE between two snapshots (the healthy sign).
|
||||
let d = late.overallMean - early.overallMean
|
||||
if d > 0.05: "rising"
|
||||
elif d < -0.05: "falling"
|
||||
else: "flat"
|
||||
|
||||
# ── state histogram (the raw picture behind settledness) ─────────────────────
|
||||
|
||||
type
|
||||
StateHistogram* = object
|
||||
nStates*: int
|
||||
nBins*: int
|
||||
binLo*: seq[int] ## inclusive low state observed in the bin
|
||||
binHi*: seq[int] ## inclusive high state observed in the bin
|
||||
counts*: seq[int]
|
||||
total*: int
|
||||
|
||||
proc stateHistogram*(m: TmMachine, nBins = 9): StateHistogram =
|
||||
## Distribution of every automaton state across [-nStates, nStates].
|
||||
let nb = max(1, nBins)
|
||||
let span = 2 * m.nStates + 1
|
||||
result.nStates = m.nStates
|
||||
result.nBins = nb
|
||||
result.counts = newSeq[int](nb)
|
||||
result.binLo = newSeq[int](nb)
|
||||
result.binHi = newSeq[int](nb)
|
||||
for b in 0..<nb:
|
||||
result.binLo[b] = high(int)
|
||||
result.binHi[b] = low(int)
|
||||
for c in 0..<m.nClasses:
|
||||
for i in 0..<m.teams[c].len:
|
||||
let st = int(m.teams[c][i])
|
||||
var bin = ((st + m.nStates) * nb) div span
|
||||
if bin < 0: bin = 0
|
||||
if bin >= nb: bin = nb - 1
|
||||
inc result.counts[bin]
|
||||
if st < result.binLo[bin]: result.binLo[bin] = st
|
||||
if st > result.binHi[bin]: result.binHi[bin] = st
|
||||
inc result.total
|
||||
for b in 0..<nb:
|
||||
if result.counts[b] == 0:
|
||||
result.binLo[b] = 0
|
||||
result.binHi[b] = 0
|
||||
|
||||
proc histogramText*(h: StateHistogram): string =
|
||||
## One line per bin, e.g. `[-64..-57] 1234 #####`.
|
||||
for b in 0..<h.nBins:
|
||||
let bar = "#".repeat(min(40, h.counts[b] div max(1, h.total div 200 + 1)))
|
||||
result.add &" [{h.binLo[b]:>4}..{h.binHi[b]:>4}] {h.counts[b]:>8} {bar}\n"
|
||||
|
||||
# ── per-input automata confidence table ──────────────────────────────────────
|
||||
|
||||
type
|
||||
InputConfidence* = object
|
||||
bit*: int
|
||||
name*: string
|
||||
nAutomata*: int
|
||||
meanCommitment*: float
|
||||
settledFraction*: float
|
||||
constant*: bool ## zero variance across the sample set (information-free)
|
||||
|
||||
proc perInputConfidence*(m: TmMachine, spec: FeatureSpec,
|
||||
samples: openArray[DiagSample],
|
||||
threshold = 0.5): seq[InputConfidence] =
|
||||
## For every raw bit: the mean commitment of the automata of its positive
|
||||
## literal and its negation, across every clause and class. Inputs the TM is
|
||||
## confident about sit high; a bit whose automata are all parked at the
|
||||
## boundary (e.g. a constant input) sits near 0. `constant` flags zero-variance
|
||||
## bits so a high commitment on a never-written input is not mistaken for
|
||||
## learning (the shipped gun's bits 38/39 are the canonical example).
|
||||
result = newSeq[InputConfidence](m.nBits)
|
||||
var consts: seq[int]
|
||||
if samples.len > 0:
|
||||
for b in 0..<m.nBits:
|
||||
let v0 = samples[0].lits[b]
|
||||
var cst = true
|
||||
for i in 1..<samples.len:
|
||||
if samples[i].lits[b] != v0:
|
||||
cst = false
|
||||
break
|
||||
if cst: consts.add b
|
||||
for b in 0..<m.nBits:
|
||||
result[b].bit = b
|
||||
result[b].name = spec.describe(b)
|
||||
result[b].constant = b in consts
|
||||
var sum = 0.0
|
||||
var settled = 0
|
||||
var n = 0
|
||||
for c in 0..<m.nClasses:
|
||||
for cl in 0..<m.nClauses:
|
||||
let base = cl * m.nLiterals
|
||||
let cmPos = stateCommitment(m.nStates, int(m.teams[c][base + b]))
|
||||
let cmNeg = stateCommitment(m.nStates,
|
||||
int(m.teams[c][base + b + m.nBits]))
|
||||
sum += cmPos + cmNeg
|
||||
if cmPos >= threshold: inc settled
|
||||
if cmNeg >= threshold: inc settled
|
||||
inc n, 2
|
||||
result[b].nAutomata = n
|
||||
result[b].meanCommitment = if n > 0: sum / n.float else: 0.0
|
||||
result[b].settledFraction = if n > 0: settled.float / n.float else: 0.0
|
||||
|
||||
proc rankedInputConfidence*(conf: seq[InputConfidence]): seq[InputConfidence] =
|
||||
## Least confident first — the inputs whose automata are still undecided.
|
||||
result = conf
|
||||
result.sort(proc(a, b: InputConfidence): int =
|
||||
result = cmp(a.meanCommitment, b.meanCommitment)
|
||||
if result == 0: result = cmp(a.bit, b.bit))
|
||||
|
||||
# ── 2. CLAUSE DIVERSITY ──────────────────────────────────────────────────────
|
||||
|
||||
type
|
||||
DiversityResult* = object
|
||||
jaccardPos*: float ## mean pairwise Jaccard among positive clauses
|
||||
jaccardNeg*: float ## ... among negative clauses
|
||||
jaccardOverall*: float ## pooled over both polarities
|
||||
nPosClauses*: int
|
||||
nNegClauses*: int
|
||||
nPairs*: int
|
||||
posValid*: bool
|
||||
negValid*: bool
|
||||
valid*: bool
|
||||
skipEmpty*: bool
|
||||
|
||||
proc jaccard*(a, b: openArray[int]): float =
|
||||
## |a n b| / |a u b| for two sorted, unique literal sets. Two empty sets are
|
||||
## identical (1.0); an empty against a non-empty set share nothing (0.0).
|
||||
if a.len == 0 and b.len == 0: return 1.0
|
||||
if a.len == 0 or b.len == 0: return 0.0
|
||||
var i = 0
|
||||
var j = 0
|
||||
var inter = 0
|
||||
while i < a.len and j < b.len:
|
||||
if a[i] == b[j]: inc inter; inc i; inc j
|
||||
elif a[i] < b[j]: inc i
|
||||
else: inc j
|
||||
let uni = a.len + b.len - inter
|
||||
if uni == 0: 1.0 else: inter.float / uni.float
|
||||
|
||||
proc polarityDiversity(m: TmMachine, polarity: int, skipEmpty: bool):
|
||||
tuple[sum: float, npairs, nclauses: int] =
|
||||
var sets: seq[seq[int]]
|
||||
let cl0 = if polarity > 0: 0 else: m.half
|
||||
let cl1 = if polarity > 0: m.half else: m.nClauses
|
||||
for c in 0..<m.nClasses:
|
||||
for cl in cl0..<cl1:
|
||||
let ls = m.clauseLits(c, cl)
|
||||
if skipEmpty and ls.len == 0: continue
|
||||
sets.add ls
|
||||
result.nclauses = sets.len
|
||||
for i in 0..<sets.len:
|
||||
for j in (i + 1)..<sets.len:
|
||||
result.sum += jaccard(sets[i], sets[j])
|
||||
inc result.npairs
|
||||
|
||||
proc clauseDiversity*(m: TmMachine, skipEmpty = true): DiversityResult =
|
||||
## Mean pairwise Jaccard overlap of the INCLUDED-literal sets, within the same
|
||||
## polarity (positive vs positive, negative vs negative). Low-to-moderate is
|
||||
## healthy; near 1.0 means the clauses are the same rule wearing 50 hats;
|
||||
## near 0 with few samples usually means memorising individual ticks. Empty
|
||||
## clauses carry no rule and are skipped by default (`skipEmpty`).
|
||||
result.skipEmpty = skipEmpty
|
||||
let p = polarityDiversity(m, 1, skipEmpty)
|
||||
let n = polarityDiversity(m, -1, skipEmpty)
|
||||
result.nPosClauses = p.nclauses
|
||||
result.nNegClauses = n.nclauses
|
||||
result.posValid = p.npairs > 0
|
||||
result.negValid = n.npairs > 0
|
||||
result.jaccardPos = if p.npairs > 0: p.sum / p.npairs.float else: 0.0
|
||||
result.jaccardNeg = if n.npairs > 0: n.sum / n.npairs.float else: 0.0
|
||||
let pairs = p.npairs + n.npairs
|
||||
result.nPairs = pairs
|
||||
result.valid = pairs > 0
|
||||
result.jaccardOverall =
|
||||
if pairs > 0: (p.sum + n.sum) / pairs.float else: 0.0
|
||||
|
||||
# ── 3. CHURN ─────────────────────────────────────────────────────────────────
|
||||
|
||||
type
|
||||
ChurnResult* = object
|
||||
samples*: int
|
||||
epochs*: int
|
||||
nAutomata*: int
|
||||
nClauses*: int
|
||||
totalFlips*: int
|
||||
totalClauseChanges*: int
|
||||
flipRate*: float ## automaton flips per training sample (fraction)
|
||||
flipRatePer100*: float ## same, x100 (readable)
|
||||
clauseChangeRate*: float ## clause-set changes per training sample
|
||||
clauseChangePer100*: float
|
||||
flipFirst*: float
|
||||
flipLast*: float
|
||||
clauseFirst*: float
|
||||
clauseLast*: float
|
||||
flipTrend*: string
|
||||
clauseTrend*: string
|
||||
windowSize*: int
|
||||
flipWindows*: seq[float] ## mean per-sample flip rate per window
|
||||
clauseWindows*: seq[float]
|
||||
trained*: TmMachine
|
||||
|
||||
proc binMeans*(s: seq[float], window: int): seq[float] =
|
||||
if s.len == 0: return
|
||||
let w = max(1, window)
|
||||
var i = 0
|
||||
while i < s.len:
|
||||
let j = min(s.len, i + w)
|
||||
var sum = 0.0
|
||||
for k in i..<j: sum += s[k]
|
||||
result.add sum / (j - i).float
|
||||
i = j
|
||||
|
||||
proc firstLastMeans*(s: seq[float]): tuple[first, last: float] =
|
||||
if s.len == 0: return (0.0, 0.0)
|
||||
let q = max(1, s.len div 5)
|
||||
var a = 0.0
|
||||
var b = 0.0
|
||||
for i in 0..<q: a += s[i]
|
||||
for i in (s.len - q)..<s.len: b += s[i]
|
||||
(a / q.float, b / q.float)
|
||||
|
||||
proc classifyChurnTrend*(s: seq[float]): string =
|
||||
## "falling" is the healthy sign; "flat"/"rising" at a high rate is
|
||||
## fidgeting; "frozen" means it never moved at all.
|
||||
if s.len < 4: return "insufficient"
|
||||
let (first, last) = firstLastMeans(s)
|
||||
if first <= 1e-12 and last <= 1e-12: return "frozen"
|
||||
if first <= 1e-12: return "rising"
|
||||
let ratio = last / first
|
||||
if ratio < 0.6: "falling"
|
||||
elif ratio > 1.6: "rising"
|
||||
else: "flat"
|
||||
|
||||
proc churnTrace*(tmpl: TmMachine, samples: openArray[DiagSample],
|
||||
epochs = 1, seed = 777'u64, window = 100,
|
||||
shuffle = true): ChurnResult =
|
||||
## Train a fresh machine while measuring, per training sample, both churn
|
||||
## levels: the fraction of AUTOMATA crossing the flip boundary, and the
|
||||
## fraction of CLAUSES whose included-literal set changed. `shuffle = false`
|
||||
## replays the samples in their given (temporal) order once per epoch, which
|
||||
## mirrors a live gun training as bullets resolve.
|
||||
result.epochs = max(1, epochs)
|
||||
result.samples = samples.len
|
||||
result.windowSize = max(1, window)
|
||||
var m = tmpl
|
||||
m.resetMachine(seed)
|
||||
result.nAutomata = m.nClasses * m.nClauses * m.nLiterals
|
||||
result.nClauses = m.nClasses * m.nClauses
|
||||
if result.nAutomata == 0:
|
||||
result.trained = m
|
||||
return
|
||||
var prevInc = newSeq[uint8](result.nAutomata)
|
||||
for c in 0..<m.nClasses:
|
||||
let team = m.teams[c]
|
||||
let gbase = c * m.nClauses * m.nLiterals
|
||||
for i in 0..<team.len:
|
||||
prevInc[gbase + i] = uint8(team[i] > 0)
|
||||
var flipSeries = newSeq[float]()
|
||||
var clauseSeries = newSeq[float]()
|
||||
for _ in 0..<result.epochs:
|
||||
let idx =
|
||||
if shuffle: shuffledIndices(samples.len, m.rng)
|
||||
else: (block:
|
||||
var id = newSeq[int](samples.len)
|
||||
for i in 0..<samples.len: id[i] = i
|
||||
id)
|
||||
for k in idx:
|
||||
m.trainSample(samples[k].lits, samples[k].label)
|
||||
var flips = 0
|
||||
var clauseChanges = 0
|
||||
for c in 0..<m.nClasses:
|
||||
let team = m.teams[c]
|
||||
let gbase = c * m.nClauses * m.nLiterals
|
||||
for cl in 0..<m.nClauses:
|
||||
var changed = false
|
||||
let base = cl * m.nLiterals
|
||||
let gb = gbase + base
|
||||
for lit in 0..<m.nLiterals:
|
||||
let nowInc = uint8(team[base + lit] > 0)
|
||||
if nowInc != prevInc[gb + lit]:
|
||||
inc flips
|
||||
changed = true
|
||||
prevInc[gb + lit] = nowInc
|
||||
if changed: inc clauseChanges
|
||||
result.totalFlips += flips
|
||||
result.totalClauseChanges += clauseChanges
|
||||
flipSeries.add flips.float / result.nAutomata.float
|
||||
clauseSeries.add clauseChanges.float / result.nClauses.float
|
||||
result.trained = m
|
||||
let nObs = flipSeries.len
|
||||
if nObs > 0:
|
||||
result.flipRate = result.totalFlips.float /
|
||||
(nObs.float * result.nAutomata.float)
|
||||
result.clauseChangeRate = result.totalClauseChanges.float /
|
||||
(nObs.float * result.nClauses.float)
|
||||
result.flipRatePer100 = result.flipRate * 100.0
|
||||
result.clauseChangePer100 = result.clauseChangeRate * 100.0
|
||||
let ff = firstLastMeans(flipSeries)
|
||||
let cf = firstLastMeans(clauseSeries)
|
||||
result.flipFirst = ff.first
|
||||
result.flipLast = ff.last
|
||||
result.clauseFirst = cf.first
|
||||
result.clauseLast = cf.last
|
||||
result.flipTrend = classifyChurnTrend(flipSeries)
|
||||
result.clauseTrend = classifyChurnTrend(clauseSeries)
|
||||
result.flipWindows = binMeans(flipSeries, result.windowSize)
|
||||
result.clauseWindows = binMeans(clauseSeries, result.windowSize)
|
||||
|
||||
# ── 4. VOTE DISAGREEMENT ─────────────────────────────────────────────────────
|
||||
|
||||
type
|
||||
VoteDisagreementResult* = object
|
||||
overall*: float
|
||||
perClass*: seq[float]
|
||||
perClassSamples*: seq[int]
|
||||
nPairs*: int ## (class, sample) observations that had a vote
|
||||
|
||||
proc voteDisagreement*(m: TmMachine, samples: openArray[DiagSample]):
|
||||
VoteDisagreementResult =
|
||||
## For each class, over the samples where it casts at least one clause vote:
|
||||
## the fraction of FIRING clauses whose own polarity disagrees with the sign
|
||||
## of the class's total vote. A class whose clauses always pull together has
|
||||
## disagreement ~0; a fuzzy class boundary or too few clauses makes it high.
|
||||
## A tied class vote (mixed signs) counts every firing clause as disagreeing.
|
||||
result.perClass = newSeq[float](m.nClasses)
|
||||
result.perClassSamples = newSeq[int](m.nClasses)
|
||||
var classSum = newSeq[float](m.nClasses)
|
||||
for c in 0..<m.nClasses:
|
||||
for s in samples:
|
||||
var vote = 0.0
|
||||
var firing: seq[int]
|
||||
for cl in 0..<m.nClauses:
|
||||
let o = m.tmEval(m.teams[c], s.lits, cl, learning = false)
|
||||
if o == 1'u8:
|
||||
vote += m.tmPolarity(cl)
|
||||
firing.add cl
|
||||
if firing.len == 0: continue
|
||||
let majSign = if vote > 0.0: 1 elif vote < 0.0: -1 else: 0
|
||||
var disagree = 0
|
||||
for cl in firing:
|
||||
let pol = if m.tmPolarity(cl) > 0.0: 1 else: -1
|
||||
if pol != majSign: inc disagree
|
||||
classSum[c] += disagree.float / firing.len.float
|
||||
inc result.perClassSamples[c]
|
||||
inc result.nPairs
|
||||
var totalSum = 0.0
|
||||
var totalN = 0
|
||||
for c in 0..<m.nClasses:
|
||||
if result.perClassSamples[c] > 0:
|
||||
result.perClass[c] = classSum[c] / result.perClassSamples[c].float
|
||||
totalSum += result.perClass[c]
|
||||
inc totalN
|
||||
result.overall = if totalN > 0: totalSum / totalN.float else: 0.0
|
||||
|
||||
# ── the convenience driver + the one-line health summary ─────────────────────
|
||||
|
||||
type
|
||||
AutomataDiag* = object
|
||||
machine*: TmMachine
|
||||
settledness*: SettlednessResult
|
||||
diversity*: DiversityResult
|
||||
churn*: ChurnResult
|
||||
disagreement*: VoteDisagreementResult
|
||||
histogram*: StateHistogram
|
||||
inputConfidence*: seq[InputConfidence]
|
||||
summary*: string
|
||||
|
||||
proc settlednessVerdict*(s: SettlednessResult): string =
|
||||
if s.overallMean >= 0.65: "settled"
|
||||
elif s.overallMean >= 0.40: "settling"
|
||||
else: "wavering"
|
||||
|
||||
proc diversityVerdict*(d: DiversityResult): string =
|
||||
if not d.valid: "n/a"
|
||||
elif d.jaccardOverall < 0.15: "diverse"
|
||||
elif d.jaccardOverall <= 0.70: "moderate"
|
||||
else: "redundant"
|
||||
|
||||
proc churnVerdict*(c: ChurnResult): string =
|
||||
case c.flipTrend
|
||||
of "falling": "settling"
|
||||
of "rising": "fidgeting (rising)"
|
||||
of "frozen": "frozen"
|
||||
of "insufficient", "not-measured": c.flipTrend
|
||||
else:
|
||||
if c.flipRatePer100 > 5.0: "fidgeting" else: "quiet"
|
||||
|
||||
proc disagreementVerdict*(d: VoteDisagreementResult): string =
|
||||
if d.overall < 0.15: "coherent"
|
||||
elif d.overall < 0.35: "fuzzy"
|
||||
else: "fragmented"
|
||||
|
||||
proc healthLine*(d: AutomataDiag): string =
|
||||
## ONE line to glance at: settledness / diversity / churn trend / disagreement,
|
||||
## each with its own verdict word.
|
||||
&"settledness={d.settledness.overallMean:.3f} ({settlednessVerdict(d.settledness)}) | " &
|
||||
&"diversity={d.diversity.jaccardOverall:.3f} ({diversityVerdict(d.diversity)}) | " &
|
||||
&"churn={d.churn.flipRatePer100:.3f}/100 {d.churn.flipTrend} " &
|
||||
&"({churnVerdict(d.churn)}) | " &
|
||||
&"disagreement={d.disagreement.overall:.3f} " &
|
||||
&"({disagreementVerdict(d.disagreement)})"
|
||||
|
||||
proc automataDiagnostics*(tmpl: TmMachine, samples: openArray[DiagSample],
|
||||
spec: FeatureSpec, epochs = 1, seed = 777'u64,
|
||||
settleThreshold = 0.5, nHistBins = 9,
|
||||
window = 100, measureChurn = true): AutomataDiag =
|
||||
## Train a fresh machine on `samples` (measuring churn along the way unless
|
||||
## `measureChurn` is false) and compute every automata-level metric on it.
|
||||
if measureChurn:
|
||||
result.churn = churnTrace(tmpl, samples, epochs, seed, window)
|
||||
result.machine = result.churn.trained
|
||||
else:
|
||||
result.machine = trainModel(tmpl, samples, epochs, seed)
|
||||
result.churn.epochs = max(1, epochs)
|
||||
result.churn.samples = samples.len
|
||||
result.churn.trained = result.machine
|
||||
result.churn.flipTrend = "not-measured"
|
||||
result.churn.clauseTrend = "not-measured"
|
||||
result.settledness = settledness(result.machine, settleThreshold)
|
||||
result.diversity = clauseDiversity(result.machine)
|
||||
result.disagreement = voteDisagreement(result.machine, samples)
|
||||
result.histogram = stateHistogram(result.machine, nHistBins)
|
||||
result.inputConfidence = perInputConfidence(result.machine, spec, samples,
|
||||
settleThreshold)
|
||||
result.summary = healthLine(result)
|
||||
|
||||
proc automataVerdict*(d: AutomataDiag): string =
|
||||
## One word for the whole trajectory: settling / fidgeting / collapsed / other.
|
||||
if d.churn.flipTrend == "falling" and d.settledness.overallMean >= 0.40:
|
||||
"settling"
|
||||
elif d.churn.flipTrend in ["flat", "rising"] and d.churn.flipRatePer100 > 5.0:
|
||||
"fidgeting"
|
||||
elif d.settledness.overallMean < 0.25 and d.churn.flipTrend == "frozen":
|
||||
"collapsed"
|
||||
elif d.disagreement.overall > 0.35 and d.settledness.overallMean < 0.40:
|
||||
"collapsed"
|
||||
elif d.settledness.overallMean >= 0.65 and d.churn.flipTrend == "frozen":
|
||||
"converged-static"
|
||||
else:
|
||||
"mixed"
|
||||
|
||||
proc formatAutomataReport*(d: AutomataDiag): string =
|
||||
## The full readout: health line, verdict, state histogram, per-input
|
||||
## confidence (least confident first) and per-class disagreement.
|
||||
result.add "health: " & d.summary & "\n"
|
||||
result.add "verdict: " & automataVerdict(d) & "\n"
|
||||
result.add &"automata: n={d.settledness.nAutomata} settled>={d.settledness.threshold:.2f} " &
|
||||
&"frac={d.settledness.overallSettledFraction:.3f}\n"
|
||||
result.add &"churn: flips/sample={d.churn.flipRate:.6f} ({d.churn.flipRatePer100:.3f}/100, " &
|
||||
&"{d.churn.flipTrend}: {d.churn.flipFirst:.6f}->{d.churn.flipLast:.6f}) " &
|
||||
&"clause-change/sample={d.churn.clauseChangeRate:.6f} " &
|
||||
&"({d.churn.clauseChangePer100:.3f}/100, {d.churn.clauseTrend})\n"
|
||||
result.add &"diversity: pos={d.diversity.jaccardPos:.3f} " &
|
||||
&"neg={d.diversity.jaccardNeg:.3f} overall={d.diversity.jaccardOverall:.3f} " &
|
||||
&"(pairs={d.diversity.nPairs}, posClauses={d.diversity.nPosClauses}, " &
|
||||
&"negClauses={d.diversity.nNegClauses})\n"
|
||||
result.add "state histogram:\n" & histogramText(d.histogram)
|
||||
result.add "per-input confidence (least confident first):\n"
|
||||
let ranked = rankedInputConfidence(d.inputConfidence)
|
||||
for i in 0..<min(12, ranked.len):
|
||||
let tag = if ranked[i].constant: " CONST" else: ""
|
||||
result.add &" bit{ranked[i].bit:>2} {ranked[i].name:<28} " &
|
||||
&"meanCommit={ranked[i].meanCommitment:.3f} settled={ranked[i].settledFraction:.3f}{tag}\n"
|
||||
var cbits: seq[int]
|
||||
for ic in d.inputConfidence:
|
||||
if ic.constant: cbits.add ic.bit
|
||||
result.add &"constant inputs (zero variance): {cbits}\n"
|
||||
result.add "disagreement per class:"
|
||||
for c in 0..<d.disagreement.perClass.len:
|
||||
result.add &" c{c}={d.disagreement.perClass[c]:.3f}"
|
||||
result.add "\n"
|
||||
|
||||
Reference in New Issue
Block a user