about
Auditing the Risk Claims of Distributional Reinforcement Learning (arxiv.org)
2 points by sbulaev 82 days ago | hide | past | pdf | discuss on HN

In plain words: They tested whether the return distributions these game-playing agents learn really predict risk, comparing each claim to long-run random rollouts with statistical checks. Most strong claims failed—40 to 95 percent were refuted—so the learned "risk" is a training quirk, not real randomness.

Abstract

Distributional reinforcement learning agents learn full return distributions that are increasingly read at face value: for interpretability, risk-sensitive control, and safety monitoring. We ask a question theory anticipates but that has not been measured directly: are the risk claims of a trained distributional agent true? Our audit combines a decision-relevant screening metric (the excess Wasserstein gap between the top two actions, which equals the mass by which first-order stochastic dominance is violated), ground truth from snapshot-restart Monte Carlo, and a statistical harness (permutation nulls, bootstrap refutation, FDR control) without which the audit itself manufactures false conclusions. Across QR-DQN, C51, and IQN on MinAtar (33 runs), 40-95% of the strongest claimed risk trade-offs are refuted at 95% confidence, the placement of the strongest claims is statistically indistinguishable from truth-blind, and essentially no claim is confirmable: for these agents, the learned "risk" reflects a training artifact rather than environment stochasticity. The artifact is structural (fully formed early in training, uncorrelated with final score, idiosyncratic to each seed) and appears unchanged at full-Atari scale, with every top Breakout claim of a pretrained near-state-of-the-art QR-DQN refuted. Positive controls of known magnitude confirm 96-100% of real claims (correlation 0.89-0.92): the reading measures the agents, not the audit. Acting on the heads' CVaR advice at their most-flagged states ranges from beneficial to significantly worse than chance. Neither training for risk nor ensembling removes the artifact, and recalibration passes the audit only by nullifying the claims: the head is uninformative, not merely miscalibrated. We release the toolkit and document two silent pitfalls that produced convincing but wrong audits of our own.

Hari Prasad
arXiv:2607.11607 · cs.AI, cs.LG, stat.ML · submitted Jul 13, 2026
abstract · pdf · html · 25 pages, 8 figures, 3 tables (main text); includes supplementary material

add comment on HN