Experience replay lets online reinforcement learning agents remember and reuse experiences from the past. In prior work, experience transitions were uniformly sampled from a replay memory. However, this approach simply replays transitions at the same frequency that they were originally experienced, regardless of their significance. In this paper we develop a framework for prioritizing experience, so as to replay important transitions more frequently, and therefore learn more efficiently. We use prioritized experience replay in Deep Q-Networks (DQN), a reinforcement learning algorithm that achieved human-level performance across many Atari games. DQN with prioritized experience replay achieves a new state-of-the-art, outperforming DQN with uniform replay on 41 out of 49 games.
核心贡献 · Key contributions
提出优先经验回放，更频繁地回放重要转换，提高学习效率。 Proposes prioritized experience replay to replay important transitions more frequently, improving learning efficiency.
引入随机优先级，平衡贪婪优先和均匀采样，确保多样性。 Introduces stochastic prioritization to balance greedy prioritization and uniform sampling, ensuring diversity.
使用重要性采样纠正非均匀采样引入的偏差，并采用退火调度。 Uses importance sampling to correct bias introduced by non-uniform sampling, with annealing schedule.
在 Atari 2600 基准测试中达到最先进性能，在 49 个游戏中 41 个优于 DQN 和 Double DQN。 Achieves state-of-the-art performance on Atari 2600 benchmark, outperforming DQN and Double DQN on 41 out of 49 games.
展示更快的学习速度，DQN 的中位数归一化性能从 48%提升到 106%。 Demonstrates faster learning speed, with median normalized performance increasing from 48% to 106% for DQN.
探索扩展到监督学习和离策略强化学习，展示广泛适用性。 Explores extensions to supervised learning and off-policy RL, showing broad applicability.
局限 · Limitations
基于 TD 误差的优先级在奖励有噪声或随机时可能次优。 Prioritization based on TD error may be suboptimal when rewards are noisy or stochastic.
贪婪优先级因采样转换缺乏多样性可能导致过拟合。 Greedy prioritization can lead to overfitting due to lack of diversity in sampled transitions.
重要性采样校正需要仔细退火；完全校正可能在非平稳设置中损害性能。 Importance sampling correction requires careful annealing; full correction may harm performance in non-stationary settings.
方法假设回放内存内容固定，不解决存储哪些经验的问题。 Method assumes replay memory contents are fixed; does not address which experiences to store.
超参数（α, β）需要针对领域调整；固定设置可能无法泛化到所有任务。 Hyperparameters (α, β) require tuning per domain; fixed settings may not generalize across all tasks.