HotpotQA：一个多样化、可解释的多跳问答数据集

HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering

杨植麟 Zhilin Yang · Carnegie Mellon University / Stanford / MILA · 2018-09-25 · arXiv:1809.09600 ↗ · 被引 4817

打开互动全文版（逐段中英对照 + 图/公式 + 论文问答）→

摘要 · Abstract

现有的问答数据集无法训练问答系统进行复杂推理并提供答案解释。我们引入了 HotpotQA，这是一个包含 11.3 万个基于维基百科的问答对的新数据集，具有四个关键特征：（1）问题需要查找并推理多个支持文档才能回答；（2）问题多样化，不受任何现有知识库或知识模式的约束；（3）我们提供推理所需的句子级支持事实，使问答系统能够在强监督下进行推理并解释预测；（4）我们提供了一种新类型的事实比较问题，以测试问答系统提取相关事实并进行必要比较的能力。我们表明，HotpotQA 对最新的问答系统具有挑战性，支持事实使模型能够提高性能并做出可解释的预测。

Existing question answering (QA) datasets fail to train QA systems to perform complex reasoning and provide explanations for answers. We introduce HotpotQA, a new dataset with 113k Wikipedia-based question-answer pairs with four key features: (1) the questions require finding and reasoning over multiple supporting documents to answer; (2) the questions are diverse and not constrained to any pre-existing knowledge bases or knowledge schemas; (3) we provide sentence-level supporting facts required for reasoning, allowing QA systems to reason with strong supervision and explain the predictions; (4) we offer a new type of factoid comparison questions to test QA systems' ability to extract relevant facts and perform necessary comparison. We show that HotpotQA is challenging for the latest QA systems, and the supporting facts enable models to improve performance and make explainable predictions.

核心贡献 · Key contributions

• 提出 HotpotQA，一个包含 11.3 万对基于维基百科的多跳问答数据集，问题多样。
  Introduces HotpotQA, a 113k Wikipedia-based multi-hop QA dataset with diverse questions.
• 提供句子级支持事实用于推理，使预测具有可解释性。
  Provides sentence-level supporting facts for reasoning, enabling explainable predictions.
• 包含新型事实比较问题，需要算术比较。
  Includes a novel type of factoid comparison questions requiring arithmetic comparison.
• 表明 HotpotQA 对最新问答系统具有挑战性，支持事实可提升性能。
  Shows that HotpotQA challenges latest QA systems and supporting facts improve performance.
• 提供两种基准设置：干扰项和全维基，测试检索与推理能力。
  Offers two benchmark settings: distractor and full wiki, testing retrieval and reasoning.
• 证明结合支持事实监督的多任务学习可提升问答准确率。
  Demonstrates that multi-task learning with supporting fact supervision boosts QA accuracy.

局限 · Limitations

• 数据集仅限于英文维基百科，可能无法泛化到其他语言或领域。
  Dataset limited to English Wikipedia, may not generalize to other languages or domains.
• 多跳推理类型预定义，可能未涵盖所有真实世界的推理模式。
  Multi-hop reasoning types are predefined; may not cover all real-world reasoning patterns.
• 基线模型性能远低于人类，表明仍有改进空间。
  Baseline model performance significantly lower than human, indicating room for improvement.
• 支持事实监督方法次优，需更好建模以发挥全部潜力。
  Supporting fact supervision method is suboptimal; better modeling needed for full potential.
• 全维基设置中检索性能较低，限制了端到端问答的有效性。
  Full wiki setting retrieval performance is low, limiting end-to-end QA effectiveness.

论文章节 · Sections（共 12）

• 摘要 Abstract
• 1 引言 1 Introduction
• 2 数据收集 2 Data Collection
• 3 处理与基准设置 3 Processing and Benchmark Settings
• 4 数据集分析 4 Dataset Analysis
• 5 实验 5 Experiments
• 5.1 模型架构与训练 5.1 Model Architecture and Training
• 5.2 结果 5.2 Results
• 5.3 建立人类表现基准 5.3 Establishing Human Performance
• 6 相关工作 6 Related Work
• 7 结论 7 Conclusions
• 致谢 Acknowledgements

阅读逐段中英对照全文 →