Eureka: Human-Level Reward Design via Coding Large Language Models
打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→大型语言模型（LLM）在顺序决策任务中作为高级语义规划器表现出色。然而，如何利用它们学习复杂的低级操作任务（如灵巧的笔旋转）仍然是一个开放问题。我们弥合了这一基本差距，提出了 Eureka——一种由 LLM 驱动的人类级别奖励设计算法。Eureka 利用最先进的 LLM（如 GPT-4）卓越的零样本生成、代码编写和上下文改进能力，对奖励代码进行进化优化。生成的奖励随后可通过强化学习获取复杂技能。无需任何特定任务提示或预定义奖励模板，Eureka 生成的奖励函数优于专家手工设计的奖励。在包含 10 种不同机器人形态的 29 个开源 RL 环境套件中，Eureka 在 83%的任务上优于人类专家，平均标准化改进达 52%。Eureka 的通用性还实现了一种新的无梯度上下文学习方法，用于从人类反馈中进行强化学习（RLHF），无需模型更新即可轻松融入人类输入以提高生成奖励的质量和安全性。最后，在课程学习设置中使用 Eureka 奖励，我们首次展示了模拟 Shadow Hand 能够执行笔旋转技巧，以快速速度熟练地操纵笔进行圆周运动。
Large Language Models (LLMs) have excelled as high-level semantic planners for sequential decision-making tasks. However, harnessing them to learn complex low-level manipulation tasks, such as dexterous pen spinning, remains an open problem. We bridge this fundamental gap and present Eureka, a human-level reward design algorithm powered by LLMs. Eureka exploits the remarkable zero-shot generation, code-writing, and in-context improvement capabilities of state-of-the-art LLMs, such as GPT-4, to perform evolutionary optimization over reward code. The resulting rewards can then be used to acquire complex skills via reinforcement learning. Without any task-specific prompting or pre-defined reward templates, Eureka generates reward functions that outperform expert human-engineered rewards. In a diverse suite of 29 open-source RL environments that include 10 distinct robot morphologies, Eureka outperforms human experts on 83% of the tasks, leading to an average normalized improvement of 52%. The generality of Eureka also enables a new gradient-free in-context learning approach to reinforcement learning from human feedback (RLHF), readily incorporating human inputs to improve the quality and the safety of the generated rewards without model updating. Finally, using Eureka rewards in a curriculum learning setting, we demonstrate for the first time, a simulated Shadow Hand capable of performing pen spinning tricks, adeptly manipulating a pen in circles at rapid speed.