Dealing with sparse rewards is one of the biggest challenges in Reinforcement Learning (RL). We present a novel technique called Hindsight Experience Replay which allows sample-efficient learning from rewards which are sparse and binary and therefore avoid the need for complicated reward engineering. It can be combined with an arbitrary off-policy RL algorithm and may be seen as a form of implicit curriculum. We demonstrate our approach on the task of manipulating objects with a robotic arm. In particular, we run experiments on three different tasks: pushing, sliding, and pick-and-place, in each case using only binary rewards indicating whether or not the task is completed. Our ablation studies show that Hindsight Experience Replay is a crucial ingredient which makes training possible in these challenging environments. We show that our policies trained on a physics simulation can be deployed on a physical robot and successfully complete the task.
核心贡献 · Key contributions
提出后见经验回放（HER），可从稀疏二进制奖励中高效学习，避免奖励工程。 Proposes Hindsight Experience Replay (HER) for sample-efficient learning from sparse binary rewards, avoiding reward engineering.
HER 可与任意离策略 RL 算法（如 DQN 或 DDPG）结合，充当隐式课程。 HER can be combined with any off-policy RL algorithm like DQN or DDPG, acting as implicit curriculum.
仅使用二进制成功/失败奖励，即可学习机械臂推、滑、抓取任务。 Demonstrates successful learning of robot arm tasks (pushing, sliding, pick-and-place) using only binary success/failure rewards.
仿真训练的策略可零微调迁移到真实机器人，并取得高成功率。 Policies trained in simulation transfer to a physical robot without fine-tuning, achieving high success rates.
在稀疏奖励设置中，HER 优于原始 DDPG 和基于计数的探索，即使单目标场景也是如此。 HER outperforms vanilla DDPG and count-based exploration in sparse reward settings, even with single goal.
在 HER 框架下，稀疏二进制奖励往往优于手工设计的成型奖励。 Sparse binary rewards often yield better results than hand-shaped rewards in HER framework.
局限 · Limitations
需要从状态到目标的映射，并假设环境动态与目标无关。 Requires a mapping from states to goals and assumes environment dynamics are independent of the goal.
有效性取决于回放目标的选择；次优策略可能削弱学习。 Effectiveness depends on the choice of replay goals; suboptimal strategies may impair learning.
性能会因观测噪声下降；训练时需额外注入噪声以提高鲁棒性。 Performance degrades with noisy observations; additional noise injection during training needed for robustness.
依赖离策略 RL 算法；未经修改不适用于在策略方法。 Relies on off-policy RL algorithms; not applicable to on-policy methods without modification.
在需要复杂时间依赖或中间状态无意义的任务中可能失败。 May fail in tasks requiring complex temporal dependencies or where achieving intermediate states is not meaningful.
论文章节 · Sections（共 15）
摘要Abstract
1 引言1 Introduction
2 背景2 Background
2.1 强化学习2.1 Reinforcement Learning
2.3 深度确定性策略梯度（DDPG）2.3 Deep Deterministic Policy Gradients (DDPG)
3.1 A motivating example3.1 A motivating example
3.2 多目标强化学习3.2 Multi-goal RL
3.3 算法3.3 Algorithm
4 实验4 Experiments
4.1 环境4.1 Environments
4.2 Does HER improve performance?4.2 Does HER improve performance?
4.4 HER 如何与奖励塑形交互？4.4 How does HER interact with reward shaping?