Exploration noise #10

New Issue

2026-08-16T11:51:25+02:00

SirStone commented

2026-08-16 11:51:25 +02:00

Question

Does PPO need explicit exploration noise on top of the stochastic policy, or is entropy in the action distribution sufficient?

Options:

No noise — PPO's stochastic policy (Gaussian with learned std) provides exploration naturally. Entropy bonus in the loss prevents premature convergence.
Ornstein-Uhlenbeck noise — temporally correlated, common in DDPG but unusual in PPO.
Action-space noise with decay schedule — start noisy, anneal to pure policy.

If entropy bonus is enough, what coefficient? If explicit noise, what kind and decay?

## Question Does PPO need explicit exploration noise on top of the stochastic policy, or is entropy in the action distribution sufficient? Options: - No noise — PPO's stochastic policy (Gaussian with learned std) provides exploration naturally. Entropy bonus in the loss prevents premature convergence. - Ornstein-Uhlenbeck noise — temporally correlated, common in DDPG but unusual in PPO. - Action-space noise with decay schedule — start noisy, anneal to pure policy. If entropy bonus is enough, what coefficient? If explicit noise, what kind and decay?

SirStone added the wayfinder:grilling label 2026-08-16 11:51:25 +02:00

SirStone referenced this issue

2026-08-16 11:51:33 +02:00

Bot scaffold and build setup #11

SirStone referenced this issue

2026-08-16 11:52:05 +02:00

RL Bot — Wayfinder Map #1

SirStone commented

2026-08-16 12:54:48 +02:00

Resolution

No explicit exploration noise. PPO's stochastic Gaussian policy (learned log-std per action dimension, decided in #9) provides exploration naturally. Entropy bonus in the PPO loss prevents premature convergence.

Log-std floor: log_std = max(log_std, -3.0) (std ≈ 0.05 minimum). Prevents total policy collapse when switching opponents — the bot always retains a minimum level of randomness. Cheap safety net; multi-opponent curriculum training is the real fix (remains in fog).

Entropy bonus coefficient: 0.01 (standard PPO default). Tunable hyperparameter.

What was ruled out:

Ornstein-Uhlenbeck noise — DDPG technique, fights the learned std in PPO
Action-space noise with decay schedule — redundant with learned log-std
Opponent recognition / fingerprinting — not in scope; the bot adapts within a battle via the 5-tick enemy history window in the state vector, not via explicit opponent identification

## Resolution **No explicit exploration noise.** PPO's stochastic Gaussian policy (learned log-std per action dimension, decided in #9) provides exploration naturally. Entropy bonus in the PPO loss prevents premature convergence. **Log-std floor:** `log_std = max(log_std, -3.0)` (std ≈ 0.05 minimum). Prevents total policy collapse when switching opponents — the bot always retains a minimum level of randomness. Cheap safety net; multi-opponent curriculum training is the real fix (remains in fog). **Entropy bonus coefficient:** 0.01 (standard PPO default). Tunable hyperparameter. **What was ruled out:** - Ornstein-Uhlenbeck noise — DDPG technique, fights the learned std in PPO - Action-space noise with decay schedule — redundant with learned log-std - Opponent recognition / fingerprinting — not in scope; the bot adapts within a battle via the 5-tick enemy history window in the state vector, not via explicit opponent identification

SirStone closed this issue

2026-08-16 12:54:53 +02:00

SirStone referenced this issue

2026-08-16 12:56:47 +02:00

Learning rate and optimizer hyperparameters #12

SirStone referenced this issue

2026-08-16 16:59:44 +02:00

RL Bot — Wayfinder Map #1

Sign in to join this conversation.