使用自然语言模式对预训练语言模型进行提示已被证明对自然语言理解有效。然而，我们的初步研究表明，手动离散提示往往导致性能不稳定——例如，改变提示中的一个单词可能导致性能大幅下降。我们提出了一种新颖的方法 P-Tuning，它将可训练的连续提示嵌入与离散提示连接起来。实验表明，P-Tuning 不仅通过最小化各种离散提示之间的差距来稳定训练，而且在包括 LAMA 和 SuperGLUE 在内的广泛 NLU 任务上显著提升了性能。P-Tuning 在冻结和微调的语言模型上，以及在完全监督和少样本设置下都普遍有效。
Prompting a pretrained language model with natural language patterns has been proved effective for natural language understanding (NLU). However, our preliminary study reveals that manual discrete prompts often lead to unstable performance -- e.g., changing a single word in the prompt might result in substantial performance drop. We propose a novel method P-Tuning that employs trainable continuous prompt embeddings in concatenation with discrete prompts. Empirically, P-Tuning not only stabilizes training by minimizing the gap between various discrete prompts, but also improves performance by a sizeable margin on a wide range of NLU tasks including LAMA and SuperGLUE. P-Tuning is generally effective for both frozen and tuned language models, under both the fully-supervised and few-shot settings.
核心贡献 · Key contributions
提出 P-Tuning，使用可训练的连续提示嵌入与离散提示结合，以稳定并提升 NLU 性能。 Proposes P-Tuning, using trainable continuous prompt embeddings with discrete prompts to stabilize and improve NLU performance.
表明 P-Tuning 减少了不同离散提示间的性能差异，增强了训练稳定性。 Shows P-Tuning reduces performance variance across different discrete prompts, enhancing training stability.
在 LAMA 知识探测（比手动提示提升 20+点）和 SuperGLUE 的全监督及少样本设置下均取得显著提升。 Achieves significant gains on LAMA knowledge probing (20+ points over manual prompts) and SuperGLUE under both full-supervised and few-shot settings.
证明对冻结和微调的语言模型均有效，且与现有离散提示兼容。 Demonstrates effectiveness for both frozen and tuned language models, and compatibility with existing discrete prompts.
引入提示编码器（LSTM/MLP）建模连续提示嵌入间的依赖关系，提升性能。 Introduces a prompt encoder (LSTM/MLP) to model dependencies among continuous prompt embeddings, improving performance.
分析提示令牌的位置和数量，为最优配置提供实用指南。 Provides analysis on prompt token location and number, offering practical guidelines for optimal configuration.
局限 · Limitations
P-Tuning 需要额外的可训练参数用于连续提示，增加了模型复杂度。 P-Tuning requires additional trainable parameters for continuous prompts, increasing model complexity.
在高资源任务（如 WiC、MultiRC）上的性能提升不如低资源任务显著。 Performance gains are less pronounced on high-resource tasks (e.g., WiC, MultiRC) compared to low-resource tasks.
该方法需要搜索提示模式和令牌数量，可能因任务而异且耗时。 The method involves searching over prompt patterns and token numbers, which may be task-specific and time-consuming.
有效性依赖于提示编码器（LSTM/MLP/EMB）的选择，EMB 在某些任务上不稳定。 Effectiveness depends on the choice of prompt encoder (LSTM/MLP/EMB), with EMB being unstable on some tasks.
不直接适用于没有离散提示的任务（如 SuperGLUE 中的 ReCoRD）。 Not directly applicable to tasks without discrete prompts (e.g., ReCoRD in SuperGLUE).
论文章节 · Sections（共 12）
摘要Abstract
1 引言1 Introduction
2.1 离散提示的问题2.1 Issues with Discrete Prompts
2.2 P-Tuning2.2 P-Tuning
2.3 提示编码器2.3 Prompt Encoder
3 实验3 Experiments
3.1 知识探测3.1 Knowledge Probing
3.2 全监督学习3.2 Fully-supervised Learning
3.3 少样本学习3.3 Few-Shot Learning
3.4 稳定语言模型适应3.4 Stabilizing Language Model Adaptation