Constructing agents with planning capabilities has long been one of the main challenges in the pursuit of artificial intelligence. Tree-based planning methods have enjoyed huge success in challenging domains, such as chess and Go, where a perfect simulator is available. However, in real-world problems the dynamics governing the environment are often complex and unknown. In this work we present the MuZero algorithm which, by combining a tree-based search with a learned model, achieves superhuman performance in a range of challenging and visually complex domains, without any knowledge of their underlying dynamics. MuZero learns a model that, when applied iteratively, predicts the quantities most directly relevant to planning: the reward, the action-selection policy, and the value function. When evaluated on 57 different Atari games - the canonical video game environment for testing AI techniques, in which model-based planning approaches have historically struggled - our new algorithm achieved a new state of the art. When evaluated on Go, chess and shogi, without any knowledge of the game rules, MuZero matched the superhuman performance of the AlphaZero algorithm that was supplied with the game rules.
核心贡献 · Key contributions
提出 MuZero 算法，将基于树的搜索与学习模型结合，无需环境动态知识即可规划。 Proposes MuZero, combining tree-based search with a learned model for planning without environment dynamics.
在围棋、国际象棋、将棋中达到超人类水平，并在 57 款 Atari 游戏中取得最优结果。 Achieves superhuman performance in Go, chess, shogi, and state-of-the-art on 57 Atari games.
学习直接预测与规划相关的奖励、策略和价值函数的模型。 Learns a model predicting reward, policy, and value function directly relevant to planning.
引入价值等价模型学习，无需重建观测或真实状态。 Introduces value-equivalent model learning without reconstructing observations or true state.
证明学习模型在围棋中规划的可扩展性，与完美模型性能匹配。 Demonstrates scalability of planning with learned model matching perfect model in Go.
表明基于搜索的策略改进比无模型 Q-learning 提供更强的学习信号。 Shows search-based policy improvement provides stronger learning signal than model-free Q-learning.
局限 · Limitations
模型为确定性，随机转换扩展留待未来工作。 Model is deterministic; extension to stochastic transitions left for future work.
Atari 中性能随搜索模拟趋于平稳，表明模型不准确性。 Performance in Atari plateaus with search simulations, suggesting model inaccuracy.
训练需要大量算力（100 万小批量，棋盘游戏 800 次模拟）。 Requires significant compute for training (1 million mini-batches, 800 simulations in board games).
不直接适用于不完全信息游戏如扑克。 Not directly applicable to imperfect information games like Poker.
限于离散动作和奖励环境，未涉及连续域。 Limited to environments with discrete actions and rewards; continuous domains not addressed.