Wayfinder: Shared integration test framework for garages #126
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem Statement
Each bot garage in the monorepo develops and trains bots independently, but there's no way to run automated integration tests — actual battles against controlled opponents with assertions on outcomes. The only testing today is unit-level (assert-based
test_*.nimon pure functions). Validating that a bot behaves correctly in a battle requires manually running the Java battle runner, eyeballing stdout, and mentally checking results. Every garage that wants integration tests would have to reinvent this orchestration from scratch.Solution
A shared integration test framework in
common_libs/test_framework/that any garage can import to run battle simulation tests vianimble test. The framework wraps the existing Java battle runner, compiles bots on the fly, manages the server lifecycle, and returns structuredBattleResultobjects that testers assert on however they see fit.A tester writes a
.nimfile that callsrunBattle(bots=@["path/to/my_bot.nim", "path/to/adversary.nim"], rounds=3), gets back aBattleResult, and uses standard assertions. The framework handles everything else: compilation, server startup, bot launching, output parsing, cleanup.User Stories
nimble testin my garage and have both unit tests and integration battle tests execute, so that I get full coverage in one command.runBattle()to compile all bot sources for me, so that I don't need manual pre-compilation steps.BattleResultobject returned fromrunBattle(), so that I can assert on winners, scores, round counts, and per-bot results without parsing raw text.tests/, so that test scenarios stay organized.runBattle()without distinguishing "main" from "adversary", so that the API stays simple and flexible.config.nimspath addition and an import, so that adoption is trivial.Implementation Decisions
common_libs/test_framework/, wired into garages viaconfig.nims --path— same pattern asradar_lock.tools/battle_runner/. Server launched with--enable-initial-position(-I) flag. No mock server — the real Java server is the only backend.runBattle()call starts the server process; it stays alive for the process lifetime and is killed on exit. No per-test server restarts.runBattle()compiles all provided.nimbot sources before launching them. Compilation uses the Nim compiler directly. Errors surface as test failures with compiler output.initialPositionwhen specified by the tester.runBattle()API: Takes a flat list of bot source paths, optional round count, optional per-bot initial positions. Returns aBattleResult. No main/adversary distinction in the API signature.BattleResulttype: Structured object parsed from the Java battle runner's stdout. Contains: round results (winner, turn count), final rankings, per-bot scores. Exact fields derived from the server's output format (seeRunBattle.java).parseServerOutput(stdout: string): BattleResult. The Java runner prints a known format:=== GAME STARTED ===, per-tick lines,=== ROUND N ENDED ===,=== RESULTS ===with ranked scores.assertorunittestcheckboth work.tests/dir (e.g.tests/avoidance/). Contains: the test.nimfile, adversary.nimfiles, any test-specific bot variants. The garage's main bot source stays insrc/.nimble testruns alltests/test_*.nimandtests/*/test_*.nimfiles. Integration tests are just test files that happen to import the framework and callrunBattle().Testing Decisions
parseServerOutput): Pure function, unit-testable with captured server output strings. This is where parsing bugs concentrate. Tested with atest_parser.nimin the framework's owntests/dir.runBattle()end-to-end: The example integration test (Ticket #131) validates the full lifecycle — compile, server start, battle, parse, result. IfrunBattle()works, the internal plumbing (server manager, bot launcher) works.runBattle()seam covers them.common_libs/radar_lock/tests/test_radar_lock.nimusesunittestwithsuite/test/check. PPO_Bot tests use bareassert-stylechecktemplate. Either pattern works for framework consumers.runBattle(), gets aBattleResult, asserts on fields. Never assert on internal framework state (server PID, temp file paths, compilation commands).Out of Scope
Grilling Session — Revised Decisions
The following decisions were made during a design review (grilling session) that uncovered gaps between the original spec and the actual codebase/APIs. These supersede any conflicting statements above.
Architecture Changes
External server, not embedded. The original spec assumed
BattleRunner.create { embeddedServer() }could support initial positions. Investigation revealed that initial positions require the server to be started with the-I(--enable-initial-position) CLI flag, which is incompatible with embedded server mode. The framework starts an external server process with-I --tps -1(unlimited speed, no GUI rendering). TheBattleRunnerconnects to it in external server mode. Battles run at full computational speed — no rendering overhead.RunBattle.java must be generalized. The existing
tools/battle_runner/RunBattle.javais a custom diagnostic harness hardcoded for PPO_Bot vs Target (1 round, no CLI flags). It must be generalized into a CLI tool accepting:--server ws://host:port --bots dir1 dir2 ... --rounds N [output flags]. It stays intools/battle_runner/.Bot launch model: server-launched.
BotEntry.of(directory)works in both embedded and external server modes. The Booter launches bot processes locally regardless of server mode. The framework produces bot directories (compiled binary + JSON config + boot script); the runner discovers them viaBotEntry.Server Lifecycle
runBattle()call starts the server, stays alive for process lifetime.addQuitProckills server process. Bot processes die with the runner/booter.TANK_ROYALE_JARenv var (existing convention).Runner Output Design
Stdout/stderr split: results go to stdout (parsed by framework), diagnostics go to stderr (passthrough to terminal).
Always-on output (stdout):
=== GAME STARTED ===,=== ROUND N ENDED (turn T) ===,=== RESULTS (N rounds) ===Fine-grained diagnostic flags (stderr):
--positions--radar--scanned--bullet-fired--bullet-hit--hit-by-bullet--hit-bot--bot-death--allNo group flags in v1. Add when someone asks.
BattleResult Type
Full score breakdown parsed from results:
rank,name,totalScore,survival,lastSurvivorBonus,bulletDamage,bulletKillBonus,ramDamage,ramKillBonus,firstPlaces,secondPlaces,thirdPlacesseq[RoundResult]withroundNumber,turnCount,winnerBot Compilation
nim cinvoked from garage root —config.nims(with--path:"../common_libs") applies automatically.--pathflags needed inrunBattle()for test bots under the garage tree.gameTypes: ["classic"], version"1.0.0", one-liner boot script. Temp dir, cleaned up after test.initialPositionwritten into generated JSON config when specified by tester.runBattle() API
outputFlags: seq[string]— raw passthrough of diagnostic flags to the Java runner.initialPositionparameter kept (functional via external server with-I).Test Discovery
tests/t*.nimat top level).tests/tall.nimentry point that imports all test modules including subdirectory ones.nimble testdiscoverstall.nim.Framework Module Structure
Two source files, not four. Parser is isolated (pure, unit-testable). Server manager and bot compiler are glue code inlined in
test_framework.nim.Error Handling
runBattle(), kills battle, test fails with timeout message.Risks
BotEntry.of()in external mode — confirmed via docs but untested in this repo. #131 is the real proof.tall.nimentry point.Further Notes
⚠️ See 'Grilling Session — Revised Decisions' above for architectural changes that supersede parts of this spec.
InitialPositionfeature is a native Tank Royale debug capability. Bots declare it in their JSON config; the server respects it when launched with-I. All three coordinates (x, y, direction) are optional — omitted values randomize. This is the key enabler for deterministic test scenarios.RunBattle.javais the contract the parser depends on. If the runner's output format changes, only the parser needs updating.Grilling resolutions (from spec review)
Resolved
nim c(notnimble build) — nimble swallows output. Framework callsnim c src/bot.nimwith appropriate flags.--enable-initial-position/-Iflag) +initialPositionfield in bot JSON metadata. Server-side feature, no custom work needed.deferblocks inrunBattle()andaddExitProcfor the server singleton to kill child processes on exception/exit/signal. Prevents zombie Java/bot processes.New tickets created
Spec additions: Timeouts
All subprocess interactions must have timeouts:
runBattle()call: 120s hard ceilingTimeout values should be configurable via
runBattle()optional params with these as defaults.Dependency update
Implementation is tracked in child tickets #127–#134 (not #127–#132 as originally stated). #133 and #134 were added during spec grilling to cover Java-side modifications and BattleRunner process lifecycle.
All child tickets #127–#134 implemented in commit
57a1915. Framework is ready for use.