Recent breakthroughs in large language models (LLMs) have centered around a handful of data-rich languages. What does it take to broaden access to breakthroughs beyond first-class citizen languages? Our work introduces Aya, a massively multilingual generative language model that follows instructions in 101 languages of which over 50% are considered as lower-resourced. Aya outperforms mT0 and BLOOMZ on the majority of tasks while covering double the number of languages. We introduce extensive new evaluation suites that broaden the state-of-art for multilingual eval across 99 languages -- including discriminative and generative tasks, human evaluation, and simulated win rates that cover both held-out tasks and in-distribution performance. Furthermore, we conduct detailed investigations on the optimal finetuning mixture composition, data pruning, as well as the toxicity, bias, and safety of our models. We open-source our instruction datasets and our model at https://hf.co/CohereForAI/aya-101
核心贡献 · Key contributions
提出了 Aya，一个覆盖 101 种语言的指令微调多语言模型，其中超过一半为低资源语言。 Introduces Aya, an instruction-finetuned multilingual model covering 101 languages, over half being lower-resourced.
将多语言评估扩展到 99 种语言，涵盖判别式、生成式、人类评估和模拟胜率基准。 Expands multilingual evaluation to 99 languages with discriminative, generative, human, and simulated win-rate benchmarks.
在覆盖两倍语言的同时，在大多数任务上优于 mT0 和 BLOOMZ，缓解了多语言诅咒。 Outperforms mT0 and BLOOMZ on most tasks while covering double the languages, mitigating the curse of multilinguality.
提出了数据加权和剪枝策略，重新平衡指令微调混合数据以提升性能。 Proposes data weighting and pruning strategies that rebalance the instruction fine-tuning mixture for better performance.
引入了多语言安全上下文蒸馏，将跨语言有害生成减少 78–89%。 Introduces multilingual safety context distillation, reducing harmful generations by 78–89% across languages.
以 Apache 2.0 许可证开源模型和指令数据集，以推动进一步的多语言研究。 Open-sources the model and instruction datasets under Apache 2.0 to empower further multilingual research.
局限 · Limitations
生成仍存在语法错误、幻觉、重复和非自然结构，尤其在低资源语言中。 Generations still exhibit grammar errors, hallucinations, repetitions, and unnatural structures, especially in lower-resourced languages.
安全上下文蒸馏在通用基准上造成小幅性能下降，并可能引入错误拒绝。 Safety context distillation causes small performance drops on general benchmarks and may introduce false refusals.
毒性评估依赖于 Perspective API，其可能存在语言偏差，影响跨语言比较。 Toxicity evaluations rely on Perspective API, which may have language bias affecting cross-lingual comparisons.
训练数据仍偏向高资源语言，翻译数据可能传播系统性翻译错误。 Training data still skews toward higher-resourced languages, and translated data may propagate systematic translation errors.
模型容量限于 13B 参数；可能需要更大模型以进一步缓解多语言诅咒。 Model capacity limited to 13B parameters; larger models may be needed to further mitigate the curse of multilinguality.
论文章节 · Sections（共 30）
摘要Abstract
1 引言1 Introduction
2 数据2 Data
2.2 人类标注2.2 Human Annotations
3 实验设置3 Experimental Set-up
3.2 数据采样消融3.2 Data Sampling Ablations
3.3 基线模型3.3 Baselines
4 评估4 Evaluation
4.1 判别式任务4.1 Discriminative Tasks
4.2 生成式任务4.2 Generative Tasks
4.3 人类与 LLM 偏好评估4.3 Human and LLM Preference Evaluations
5 结果5 Results
46 语言46 Languages
52 种语言52 Languages
5.1 Discriminative Tasks5.1 Discriminative Tasks
5.2 生成任务5.2 Generative Tasks
5.3 按语言资源丰富度的性能比较5.3 Performance Comparison by Language Resourcedness
5.4 模拟胜率与人类评估5.4 Simulated Win Rates and Human Eval