Learned world models summarize an agent's experience to facilitate learning complex behaviors. While learning world models from high-dimensional sensory inputs is becoming feasible through deep learning, there are many potential ways for deriving behaviors from them. We present Dreamer, a reinforcement learning agent that solves long-horizon tasks from images purely by latent imagination. We efficiently learn behaviors by propagating analytic gradients of learned state values back through trajectories imagined in the compact state space of a learned world model. On 20 challenging visual control tasks, Dreamer exceeds existing approaches in data-efficiency, computation time, and final performance.
核心贡献 · Key contributions
提出 Dreamer，一种基于模型的强化学习智能体，完全通过在紧凑潜在空间中的潜在想象学习长时域行为。 Proposes Dreamer, a model-based RL agent that learns long-horizon behaviors purely by latent imagination in a compact latent space.
引入一种演员-评论家算法，通过学习的潜在动力学传播多步价值估计的解析梯度。 Introduces an actor-critic algorithm that propagates analytic gradients of multi-step value estimates through learned latent dynamics.
在 20 个视觉控制任务上展示最先进性能，在数据效率、计算时间和最终得分上超越先前方法。 Demonstrates state-of-the-art performance on 20 visual control tasks, exceeding prior methods in data efficiency, computation time, and final score.
表明在想象中学习价值模型使智能体对想象时域具有鲁棒性，从而能够处理长时域任务。 Shows that learning a value model in imagination makes the agent robust to the imagination horizon, enabling long-horizon tasks.
比较表示学习目标（像素重建、对比估计、奖励预测），发现像素重建最有效。 Compares representation learning objectives (pixel reconstruction, contrastive estimation, reward prediction) and finds pixel reconstruction most effective.
在离散动作任务和提前终止场景上验证 Dreamer，展示广泛适用性。 Validates Dreamer on discrete action tasks and early termination scenarios, showing broad applicability.
局限 · Limitations
依赖学习的世界模型，可能无法泛化到未见环境或高视觉复杂度。 Relies on a learned world model that may not generalize to unseen environments or high visual complexity.
需要可微动力学和重参数化采样，限制了在离散动作上的应用（需直通梯度）。 Requires differentiable dynamics and reparameterized sampling, limiting applicability to discrete actions without straight-through gradients.
性能依赖于表示学习质量；仅靠奖励预测是不够的。 Performance depends on the quality of representation learning; pure reward prediction alone was insufficient.
想象时域和价值模型超参数可能需要针对不同任务调整。 Imagination horizon and value model hyperparameters may need tuning for different tasks.
评估限于模拟环境；实际部署可能面临额外挑战。 Evaluation limited to simulated environments; real-world deployment may face additional challenges.
论文章节 · Sections（共 29）
摘要Abstract
1 引言1 Introduction
2 基于世界模型的控制2 Control with World Models
强化学习Reinforcement learning
智能体组件Agent components
潜在动力学Latent dynamics
3 通过潜在想象学习行为3 Learning Behaviors by Latent Imagination