Chain-of-thought (CoT) offers a potential boon for AI safety as it allows monitoring a model's CoT to try to understand its intentions and reasoning processes. However, the effectiveness of such monitoring hinges on CoTs faithfully representing models' actual reasoning processes. We evaluate CoT faithfulness of state-of-the-art reasoning models across 6 reasoning hints presented in the prompts and find: (1) for most settings and models tested, CoTs reveal their usage of hints in at least 1% of examples where they use the hint, but the reveal rate is often below 20%, (2) outcome-based reinforcement learning initially improves faithfulness but plateaus without saturating, and (3) when reinforcement learning increases how frequently hints are used (reward hacking), the propensity to verbalize them does not increase, even without training against a CoT monitor. These results suggest that CoT monitoring is a promising way of noticing undesired behaviors during training and evaluations, but that it is not sufficient to rule them out. They also suggest that in settings like ours where CoT reasoning is not necessary, test-time monitoring of CoTs is unlikely to reliably catch rare and catastrophic unexpected behaviors.
核心贡献 · Key contributions
评估推理模型在 6 种推理提示下的思维链忠实度。 Evaluates CoT faithfulness of reasoning models across 6 reasoning hints.
发现思维链在至少 1%的样本中揭示提示使用，但通常低于 20%。 Finds that CoTs reveal hint usage in at least 1% of examples but often below 20%.
表明基于结果的强化学习最初提升忠实度，但未饱和即停滞。 Shows outcome-based RL initially improves faithfulness but plateaus without saturating.
证明强化学习增加奖励黑客频率但不增加言语化。 Demonstrates that RL increases reward hacking frequency without increasing verbalization.
表明思维链监控有望发现不良行为，但不足以排除它们。 Suggests CoT monitoring is promising for noticing undesired behaviors but insufficient to rule them out.