In plain words: A new training update mixes a steadier way of improving an agent's choices with practice predicting what happens next in the game. It matched the best Atari agent that plans ahead with search, while acting instantly at the speed of agents that never plan.

Abstract · Muesli: Combining Improvements in Policy Optimization

We propose a novel policy update that combines regularized policy optimization with model learning as an auxiliary loss. The update (henceforth Muesli) matches MuZero's state-of-the-art performance on Atari. Notably, Muesli does so without using deep search: it acts directly with a policy network and has computation speed comparable to model-free baselines. The Atari results are complemented by extensive ablations, and by additional results on continuous control and 9x9 Go.

Matteo Hessel, Ivo Danihelka, Fabio Viola, Arthur Guez, Simon Schmitt, Laurent Sifre, Theophane Weber, David Silver, Hado van Hasselt
arXiv:2104.06159 · cs.LG, cs.AI · submitted Apr 13, 2021 · updated Mar 31, 2022
abstract · pdf · html