Despite considerable advancements with deep neural language models, the enigma of neural text degeneration persists when these models are tested as text generators. The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, using likelihood as a decoding objective leads to text that is bland and strangely repetitive. In this paper, we reveal surprising distributional differences between human text and machine text. In addition, we find that decoding strategies alone can dramatically effect the quality of machine text, even when generated from exactly the same neural language model. Our findings motivate Nucleus Sampling, a simple but effective method to draw the best out of neural generation. By sampling text from the dynamic nucleus of the probability distribution, which allows for diversity while effectively truncating the less reliable tail of the distribution, the resulting text better demonstrates the quality of human text, yielding enhanced diversity without sacrificing fluency and coherence.
核心贡献 · Key contributions
揭示了人类文本与机器文本之间令人惊讶的分布差异。 Reveals surprising distributional differences between human and machine text.
发现仅解码策略就能显著影响机器文本质量。 Finds that decoding strategies alone dramatically affect machine text quality.
提出 Nucleus Sampling，一种简单有效的解码方法。 Proposes Nucleus Sampling, a simple and effective decoding method.
表明基于最大化的解码导致文本平淡且重复。 Shows maximization-based decoding leads to bland, repetitive text.
证明 Nucleus Sampling 生成多样且连贯的文本。 Demonstrates that Nucleus Sampling yields diverse and coherent text.
使用 HUSE 评估表明 Nucleus Sampling 是最佳解码策略。 Uses HUSE evaluation to show Nucleus Sampling as the best decoding strategy.
局限 · Limitations
专注于开放式生成，而非翻译等定向任务。 Focuses on open-ended generation, not directed tasks like translation.
实验仅限于 GPT-2 Large 模型，可能不具普适性。 Experiments limited to GPT-2 Large model; may not generalize.
Nucleus Sampling 需要调整阈值 p 以获得最佳性能。 Nucleus Sampling requires tuning threshold p for optimal performance.
未解决训练阶段的文本退化问题。 Does not address training-time solutions for text degeneration.
每种方法的人类评估仅限 200 个生成样本。 Human evaluation limited to 200 generations per method.
论文章节 · Sections（共 17）
摘要Abstract
1 引言1 Introduction
2.1 文本生成解码策略2.1 Text Generation Decoding Strategies
2.2 开放式与定向生成2.2 Open-ended vs directed generation
3 语言模型解码3 Language Model Decoding
3.1 核采样3.1 Nucleus Sampling
3.2 Top-k 采样3.2 Top-k Sampling
3.3 温度采样3.3 Sampling with Temperature
4.1 实验设置4.1 Experimental Setup
4.2 困惑度4.2 Perplexity
4.3 自然语言不最大化概率4.3 Natural Language Does Not Maximize Probability
5.1 齐夫分布分析5.1 Zipf Distribution Analysis
5.2 自 BLEU5.2 Self-BLEU
5.3 重复性5.3 Repetition
6.1 人类与统计评估统一（HUSE）6.1 Human Unified with Statistical Evaluation (HUSE)