基于 TrueSkill 的 LLM 排序

LLM-Powered Sorting with TrueSkill

塔里克·希希帕尔 Thariq Shihipar · Anthropic · 2025-02-11 · Blog ↗

摘要 · Abstract

尽管大语言模型（LLM）在理解和比较概念方面表现出色，但让它们持续对大量数据进行排序仍然非常困难。

The Challenge with LLM SortingEnter TrueSkillImplementationLLM Sorting PromptAn ExampleWhen are you “done”?AlternativesIndividual ScoringEmbedding-based SortingOptimizationSmart Batch SelectionUsing Confidence IntervalsConclusion Thariq Shihipar - 11 February 2025 · 7 min read Large Language Models (LLMs) are remarkably good at understanding and comparing concepts, but getting them to consistently sort large amounts of data is still quite difficult.

核心贡献 · Key contributions

结合 LLM 语义理解与 TrueSkill 算法，实现鲁棒、可扩展的语义排序。
Combines LLM semantic understanding with TrueSkill algorithm for robust, scalable semantic sorting.
解决大规模 LLM 排序中的上下文窗口、一致性、质量下降和令牌限制问题。
Solves context window, consistency, quality degradation, and token limit issues in large-scale LLM sorting.
通过 TrueSkill 评分更新，支持增量添加新项目而无需完全重新排序。
Enables incremental addition of new items without full re-sorting via TrueSkill rating updates.
提供优化策略：智能批次选择、置信区间和混合嵌入集成。
Provides optimization strategies: smart batch selection, confidence intervals, and hybrid embedding integration.
通过提示工程展示可靠的两两比较的实际实现。
Demonstrates practical implementation with prompt engineering for reliable pairwise comparisons.

局限 · Limitations

尽管有 TrueSkill，LLM 在微妙或复杂排序标准上的一致性仍可能下降。
LLM consistency may still degrade on nuanced or complex sorting criteria despite TrueSkill.
每个项目需要多次 LLM 调用，对极大数据集可能成本高昂。
Requires many LLM calls per item, potentially high cost for very large datasets.
高度依赖提示质量；不同领域需要针对性的任务调优。
Relies heavily on prompt quality; task-specific tuning is necessary for different domains.
TrueSkill 的稳定性假设可能因 LLM 跨试验的输出差异而不成立。
TrueSkill's stability assumption may not hold due to LLM output variance across trials.
每个项目 1-2 次比赛即足够的说法缺乏严格验证，可能依赖数据集。
Claim of 1-2 matches per item being sufficient lacks rigorous validation and may be dataset-dependent.

论文章节 · Sections（共 15）

目录 CONTENTS
LLM 排序的挑战 The Challenge with LLM Sorting
TrueSkill 算法 Enter TrueSkill
实现 Implementation
初始化所有项目的评分 Initialize ratings for all items
样本批次项 Sample batch of items
更新 TrueSkill 评分 Update TrueSkill ratings
LLM 排序提示 LLM Sorting Prompt
一个示例 An Example
何时算“完成”？ When are you “done”?
个体评分 Individual Scoring
基于嵌入的排序 Embedding-based Sorting
智能批次选择 Smart Batch Selection
使用置信区间 Using Confidence Intervals
结论 Conclusion

阅读逐段中英对照全文 →