Where and what granularity should we instrument the Nim training thread with chunk-start/end logs + heartbeat to distinguish hang vs crash vs slow for corpse stall diagnosis? #68

New Issue

2026-08-25T09:45:17+02:00

SirStone commented

2026-08-25 09:45:17 +02:00

Question

Where and what granularity should we instrument the Nim training thread with chunk-start/end logs + heartbeat to distinguish hang vs crash vs slow for corpse stall diagnosis?

This is a child issue of wayfinder map #67.

## Question Where and what granularity should we instrument the Nim training thread with chunk-start/end logs + heartbeat to distinguish hang vs crash vs slow for corpse stall diagnosis? This is a child issue of wayfinder map #67.

SirStone added the wayfinder:research label 2026-08-25 09:45:17 +02:00

SirStone commented

2026-08-25 09:50:42 +02:00

Based on code examination, the corpse stall (training counter stalls while game rounds complete) likely occurs due to:

Long-running trainPass (up to 10 min) with saves only checked inside the step loop.
I/O thread blocking on gSaveChan.recv() if save takes long.
Channel backpressure causing dropped transitions.

Proposed instrumentation:

In integration.nim trainPass: log each step that takes >2000ms and log total trainPass time.
In ioThreadEntry: log save operations that take >5000ms.

This will help distinguish hang (consistently long steps/saves) from crash (no logs) from slow but acceptable.

Based on code examination, the corpse stall (training counter stalls while game rounds complete) likely occurs due to: 1. Long-running trainPass (up to 10 min) with saves only checked inside the step loop. 2. I/O thread blocking on gSaveChan.recv() if save takes long. 3. Channel backpressure causing dropped transitions. Proposed instrumentation: - In integration.nim trainPass: log each step that takes >2000ms and log total trainPass time. - In ioThreadEntry: log save operations that take >5000ms. This will help distinguish hang (consistently long steps/saves) from crash (no logs) from slow but acceptable.

SirStone closed this issue

2026-08-25 09:51:23 +02:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: SirStone/SirRoboGarage#68