Reward Models (RMs) are critical for improving generation models via Reinforcement Learning (RL), yet the RM scaling paradigm in visual generation remains largely unexplored. It primarily due to fundamental limitations in existing approaches: CLIP-based RMs suffer from architectural and input modality constraints, while prevalent Bradley-Terry losses are fundamentally misaligned with the next-token prediction mechanism of Vision-Language Models (VLMs), hindering effective scaling. More critically, the RLHF optimization process is plagued by Reward Hacking issue, where models exploit flaws in the reward signal without improving true quality. To address these challenges, we introduce RewardDance, a scalable reward modeling framework that overcomes these barriers through a novel generative reward paradigm. By reformulating the reward score as the model's probability of predicting a "yes" token, indicating that the generated image outperforms a reference image according to specific criteria, RewardDance intrinsically aligns reward objectives with VLM architectures. This alignment unlocks scaling across two dimensions: (1) Model Scaling: Systematic scaling of RMs up to 26 billion parameters; (2) Context Scaling: Integration of task-specific instructions, reference examples, and chain-of-thought (CoT) reasoning. Extensive experiments demonstrate that RewardDance significantly surpasses state-of-the-art methods in text-to-image, text-to-video, and image-to-video generation. Crucially, we resolve the persistent challenge of "reward hacking": Our large-scale RMs exhibit and maintain high reward variance during RL fine-tuning, proving their resistance to hacking and ability to produce diverse, high-quality outputs. It greatly relieves the mode collapse problem that plagues smaller models.
核心贡献 · Key contributions
提出 RewardDance，一种生成式奖励建模框架，通过下一个词预测将奖励目标与 VLM 架构对齐。 Proposes RewardDance, a generative reward modeling framework that aligns reward objectives with VLM architectures via next-token prediction.
展示了奖励模型系统性地扩展到 260 亿参数，与生成质量呈强正相关。 Demonstrates systematic scaling of reward models up to 26B parameters, showing strong positive correlation with generation quality.
引入上下文扩展，整合任务特定指令、参考示例和思维链推理。 Introduces context scaling by integrating task-specific instructions, reference examples, and chain-of-thought reasoning.
通过大规模 RM 在 RL 微调中保持高奖励方差，解决了奖励破解问题。 Resolves reward hacking by maintaining high reward variance during RL fine-tuning with large-scale RMs.
在文本到图像、文本到视频和图像到视频生成基准上取得最先进结果。 Achieves state-of-the-art results on text-to-image, text-to-video, and image-to-video generation benchmarks.
将可扩展性确立为视觉奖励模型的基本原则，为该领域提供新见解。 Establishes scalability as a foundational principle for visual reward models, providing new insights for the field.
局限 · Limitations
探索的最大模型规模为 260 亿参数；进一步扩展到 700 亿/1000 亿可能带来额外收益。 Maximum model size explored is 26B parameters; further scaling to 70B/100B may yield additional gains.
主要关注对齐；未探索运动、美学等其他维度。 Focuses primarily on alignment; other dimensions like motion and aesthetics are not explored.
任务范围限于生成；未涉及理解和编辑任务。 Task scope limited to generation; understanding and editing tasks are not addressed.
未研究视觉之外的多模态扩展（如音频到视频）。 Multimodal scaling beyond vision (e.g., audio-to-video) is not investigated.
上下文扩展限于指令、参考和 CoT；更丰富的上下文如反思和上下文学习留待未来工作。 Context scaling limited to instructions, references, and CoT; richer context like reflection and in-context learning remains future work.
论文章节 · Sections（共 17）
摘要Abstract
1 引言1 Introduction
2.1 扩散模型2.1 Diffusion Models
2.2 奖励模型2.2 Reward Models
2.3 基于人类反馈的强化学习2.3 Reinforcement Learning from Human Feedback
3 方法3 Method
3.1 预备知识3.1 Preliminary
3.2 奖励模型学习3.2 Reward Model Learning
3.3 奖励反馈对齐3.3 Reward Feedback Aligning
4 实验4 Experiments
4.1 评估基线、基准与指标4.1 Evaluation Baselines, Benchmarks and Metrics
4.2 与奖励模型规模化的比较4.2 Comparison with Reward Model Scaling
4.3 与最先进生成模型的比较4.3 Comparison with State-of-the-art Generation Models