We explore building generative neural network models of popular reinforcement learning environments. Our world model can be trained quickly in an unsupervised manner to learn a compressed spatial and temporal representation of the environment. By using features extracted from the world model as inputs to an agent, we can train a very compact and simple policy that can solve the required task. We can even train our agent entirely inside of its own hallucinated dream generated by its world model, and transfer this policy back into the actual environment. An interactive version of this paper is available at https://worldmodels.github.io/
核心贡献 · Key contributions
提出一个由 VAE 和 MDN-RNN 组成的世界模型框架，用于无监督学习压缩的时空表征。 Proposes a world model framework with VAE and MDN-RNN for unsupervised learning of compressed spatial-temporal representations.
展示利用世界模型特征训练紧凑线性控制器以解决连续控制任务。 Demonstrates training a compact linear controller using features from the world model to solve continuous control tasks.
展示智能体可在完全由世界模型生成的梦境环境中训练，并将策略迁移到真实环境。 Shows agent can be trained entirely inside a hallucinated dream environment and transfer policy to the real environment.
利用 MDN-RNN 的温度参数控制不确定性，防止对抗性策略的利用。 Uses temperature parameter in MDN-RNN to control uncertainty and prevent adversarial policy exploitation.
在 CarRacing-v0 和 VizDoom 任务上通过简单进化优化取得最先进结果。 Achieves state-of-the-art results on CarRacing-v0 and VizDoom tasks with simple evolutionary optimization.
提供论文的交互式在线版本，便于动手探索。 Provides an interactive online version of the paper for hands-on exploration.
局限 · Limitations
由于无监督训练，世界模型可能编码与任务无关的细节。 World model may encode task-irrelevant details due to unsupervised training.
基于 LSTM 的世界模型容量有限，可能导致灾难性遗忘。 Limited capacity of LSTM-based world model may cause catastrophic forgetting.
控制器可能利用世界模型的不完美，导致策略在真实环境中失败。 Controller can exploit imperfections in the world model, leading to policies that fail in the real environment.
方法依赖随机策略数据收集，对于复杂环境可能不足。 Method relies on random policy data collection, which may be insufficient for complex environments.
缺乏分层规划或抽象推理，仅逐步模拟未来。 No hierarchical planning or abstract reasoning; simulates futures step by step.
论文章节 · Sections（共 27）
摘要Abstract
1 引言1 Introduction
2 智能体模型2 Agent Model
2.1 VAE（V）模型2.1 VAE (V) Model
2.2 MDN-RNN（M）模型2.2 MDN-RNN (M) Model
2.3 控制器（C）模型2.3 Controller (C) Model
2.4 V、M 和 C 的整合2.4 Putting V, M, and C Together
3 赛车实验3 Car Racing Experiment
3.1 用于特征提取的世界模型3.1 World Model for Feature Extraction
3.2 实验步骤3.2 Procedure
3.3 实验结果3.3 Experiment Results
3.4 赛车梦境3.4 Car Racing Dreams
4.1 在梦境中学习4.1 Learning Inside of a Dream
4.2 实验步骤4.2 Procedure
4.3 在梦境中训练4.3 Training Inside of the Dream
4.4 将策略迁移到实际环境4.4 Transfer Policy to Actual Environment