Audit Finds Most 'Risk-Aware' Claims in Distributional RL Agents Don't Hold Up

A systematic audit of distributional reinforcement learning agents has found that the majority of their headline "risk-awareness" claims collapse under statistical scrutiny. The study, published on arXiv, examined QR-DQN, C51, and IQN agents trained on the MinAtar suite and discovered that between 40% and 95% of the strongest claimed risk trade-offs were refuted at 95% confidence.

Distributional RL differs from standard reinforcement learning in a specific way: instead of predicting a single expected return, these agents model an entire distribution of possible returns. That richer output is often marketed as giving the agent a sense of risk, letting it distinguish between safe, predictable strategies and volatile, high-variance ones. The audit set out to test whether that promise actually holds.

It doesn't, at least not reliably. The researchers traced the apparent "risk sensitivity" in these agents back to a training artifact rather than genuine sensitivity to randomness in the environment. Tellingly, this artifact showed up fully formed early in training, well before the agent's performance had converged, and its presence had no correlation with the agent's final score. In other words, an agent could look risk-aware from the very first stages of training regardless of whether it ever became any good at the task.

The Breakout results were especially stark. Every top-line claim involving a pretrained, near-state-of-the-art QR-DQN agent on Breakout was refuted once subjected to the audit's statistical bar. That's notable because Breakout has long served as a showcase environment for distributional RL papers demonstrating nuanced risk behavior.

To rule out the possibility that the auditing method itself was simply too strict, the team ran positive controls using effects of known magnitude. Those controls held up well, with 96% to 100% of genuine claims confirmed and correlations between 0.89 and 0.92. That suggests the audit isn't rejecting real signal indiscriminately, it's specifically catching claims that don't survive contact with proper statistical testing.

Perhaps the most damning finding involves recalibration, a common fix applied when a model's outputs are miscalibrated. When the researchers recalibrated the risk-related outputs of these agents, the claims only passed audit by becoming statistically uninformative. The recalibrated head wasn't simply adjusted to be more accurate, it effectively had nothing meaningful left to say. That distinction matters: a miscalibrated signal can potentially be corrected, but a signal that becomes null once corrected was likely never carrying the information it was assumed to carry.

For a subfield that has built research narratives, benchmark comparisons, and safety-adjacent applications around agents "understanding" risk, the findings are a pointed reminder to separate distributional richness from distributional meaning. Predicting a fuller shape of possible returns is not the same as the agent having learned anything true about the underlying uncertainty in its environment, and this audit suggests that gap has been going largely unchecked.