Diversity plays a critical role in data selection, improving performance under fixed data budgets by reducing redundancy and repetition. However, optimizing for diversity is inherently challenging, as it is a set-level property that depends on interactions between data points rather than individual examples. As a result, existing approaches typically rely on proxies or approximations, which often fail to ensure sufficiently diverse subsets. In this work, we directly optimize diversity by introducing a probabilistic diversification framework based on the G-Vendi score, optimized via exponentiated gradient descent. Our method produces subsets that are substantially more diverse than those obtained via random sampling, achieving a +489 increase in G-Vendi score on a 500k-sample subset. We evaluate our approach on FineWeb and DCLM, where it consistently outperforms existing methods. Notably, SPOKES (diversity-only) improves average downstream performance by +0.4 and +0.5 points over random sampling on DCLM and FineWeb, respectively. More importantly, jointly optimizing for both quality and diversity yields the strongest results: SPOKES achieves gains of +1.5 and +1.4 points on DCLM and FineWeb, outperforming all baselines, including semantic deduplication and quality filtering.
核心贡献 · Key contributions
提出 Spokes，一种基于 G-Vendi 分数的可扩展方法，用于预训练数据选择中的直接多样性优化。 Introduces Spokes, a scalable method for direct diversity optimization in pretraining data selection using G-Vendi score.
展示 Spokes 能提取高度多样化的子集，在 50 万样本子集上实现 G-Vendi 分数提升 489。 Shows Spokes extracts highly diverse subsets, achieving +489 G-Vendi score increase on a 500k-sample subset.
证明质量和多样性的联合优化带来一致提升，在 DCLM 和 FineWeb 上优于所有基线。 Demonstrates joint optimization of quality and diversity yields consistent gains, outperforming all baselines on DCLM and FineWeb.
通过指数梯度下降提供松弛优化，实现大规模高效子集选择。 Provides a relaxed optimization via exponentiated gradient descent, enabling efficient subset selection at scale.
实验表明基于梯度的表示比嵌入能捕捉更细粒度的多样性。 Empirically shows gradient-based representations capture finer-grained diversity than embeddings.
揭示多样性和质量编码互补的训练信号，提升跨任务的下游性能。 Reveals diversity and quality encode complementary training signals, improving downstream performance across tasks.
局限 · Limitations
需要通过代理模型计算梯度，尽管有截断，计算成本仍然高昂。 Requires gradient computation via a proxy model, which is computationally expensive despite truncation.
依赖可调参数α来平衡质量和多样性，需要针对每个数据集进行校准。 Relies on a tunable parameter α to balance quality and diversity, requiring calibration per dataset.
基于梯度的表示可能对序列长度敏感，可能混淆多样性信号。 Gradient-based representations may be sensitive to sequence length, potentially confounding diversity signals.
方法假设固定数据预算，可能不适用于动态或流式数据选择。 Method assumes a fixed data budget and may not generalize to dynamic or streaming data selection.
评估仅限于英文基准和两个语料库；对其他语言或领域的泛化性未经测试。 Evaluation limited to English benchmarks and two corpora; generalizability to other languages or domains is untested.
论文章节 · Sections（共 19）
摘要Abstract
1 引言1 Introduction
2.1 G-Vendi 作为多样性度量2.1 G-Vendi as a diversity measure
3.1 Spokes：实现高 G-Vendi 子集的可扩展优化3.1 Spokes: Scalable optimization to achieve high G-Vendi subsets
3.2 优化在质量与多样性之间实现平滑权衡3.2 The optimisation leads to smooth trade-offs between quality and diversity
3.3 Spokes 的时间复杂度3.3 Time complexity of Spokes
4 提升 Spokes 的效率4 Improving efficiency in Spokes
4.1 利用最后 nn 层近似梯度4.1 Approximating gradients using the last nn layers
4.2 子集选择中可选用更小的 k 进行 Johnson–Lindenstrauss 投影维度4.2 For subset selection a smaller k can be chosen for Johnson–Lindenstrauss projection dimensions