Language Models for Text Classification: From Bag-of-Words to Jev
打开互动全文版（逐段中英对照 + 针对本篇提问）→本文追溯了用于文本分类的语言模型的演变历程，从早期配合朴素贝叶斯和逻辑回归使用的词袋表示，到 CNN 和 RNN 等深度神经网络架构，最后到基于 Transformer 的模型，以此将近期出现的 Jev AI 分类器置于这一历史脉络中加以理解。文章解释了词袋方法将变长文本转换为固定大小的词频向量，从而实现廉价而有效的分类，但丢弃了词序信息；而词嵌入和 RNN 则以更难的训练为代价保留了序列结构。核心论点是：Jev 本质上是一个文本分类器，但又不仅仅是一个文本分类器：它比狭窄的特定任务模型更通用，又比大型通用 LLM 更快、更便宜，占据了一个实用的中间地带。对读者而言，结论是：理解这段历史有助于揭开围绕 Jev 的炒作迷雾，并厘清此类分类器真正擅长之处。
This article traces the evolution of language models for text classification, from early bag-of-words representations used with naive Bayes and logistic regression to deep neural architectures such as CNNs and RNNs, and finally to transformer-based models, in order to contextualize the recent Jev AI classifier. It explains that bag-of-words converts variable-length text into fixed-size word-frequency vectors, enabling cheap and effective classification but discarding word order, while word embeddings and RNNs preserve sequential structure at the cost of harder training. The core argument is that Jev is essentially a text classifier, yet not just a text classifier: it is more general than narrow task-specific models and much faster and cheaper than large general-purpose LLMs, occupying a practical middle ground. The conclusion for readers is that understanding this history helps demystify the hype around Jev and clarifies where such classifiers genuinely excel.
最近发布的 Jev AI 模型在过去两周内已成为技术社区中的一种文化现象。
The recently released Jev AI model has been quite a cultural phenomenon in technical communities in the past 2 weeks.
尽管 Jev 旨在对事物进行分类，但人们很容易将 Jev 贬为“仅仅是一个分类器”，而我自己对 Jev 的看法在过去几天里也发生了相当大的变化。特别是，我的想法从“分类器曾是我的拿手好戏；我可以轻松自己构建这个”（稍后详述）转变为“哇，这实际上比我预期的要好。”
While Jev aims to classify things, it’s easy to dismiss Jev as “just a classifier,” and my own view of Jev has evolved quite a bit over the past few days. In particular, my thoughts went from “classifiers used to be my bread & butter; I can easily build this myself” (more on this later) to “wow, this actually works better than I thought.”
图 1：Jev API 的快速概览；更多细节稍后介绍。
Figure 1: Quick overview of the Jev API; more details on that later.
当然，最新的最先进 GPT 和开放权重的 LLM 可以完成与 Jev 相同类型的分类任务，同时还能进行更通用的决策。但 Jev 的优势在于，它可以更快、更便宜地处理这些分类任务。
Sure, the latest state-of-the-art GPT and open-weight LLMs can do the same kinds of classification tasks as Jev, while also being capable of much more general decision-making. But Jev’s advantage is that it can handle those classification tasks much faster and more cheaply.
另一方面，对于狭窄、定义明确的问题，Jev 可能不会比专用分类器分类得更好、更快或更便宜。但它的卖点是，它比那些特定任务的模型要通用得多。
At the other end of the spectrum, for a narrow, well-defined problem, Jev probably won’t classify anything better, faster, or cheaper than a special-purpose classifier. But its selling point is that it is far more general than those task-specific models.
那么，Jev 背后的方法论是什么（基于有根据的推测），它能做什么，以及它为何如此受欢迎？我打算在本文后面回答所有这些问题。不过，我认为从决策语言模型的简要历史开始会是一个很好的开端。希望这有助于揭开一些炒作的神秘面纱，并展示 Jev 非常擅长的地方（“Jev 本质上是一个文本分类器”，但“Jev 也不‘只是’一个文本分类器”）。
So, what is the methodology behind Jev (based on an educated guess), what can it do, and why is it so popular? I aim to answer all of these later in this article. However, I thought starting with a brief history of language models for decision-making would be a great way to begin. And it hopefully helps demystify some of the hype and show what Jev does very well ("Jev is essentially a text classifier," but "Jev is also not 'just' a text classifier.").
附注：我与 Jev 没有任何隶属关系。此外，我没有获得 Jev 的免费访问权限，这也不是产品代言，只是一篇技术文章，提供一些关于文本分类历史的见解，帮助你理解最近的炒作。
PS: I am not affiliated with Jev in any way. Also, I am not offered free access to Jev, and this is also not a product endorsement, just a technical article to offer some insights into the history of text classification to help you make sense of the recent hype.
由于这是一篇长文，我建议你在浏览器中阅读，这样你可以访问左侧的目录菜单。
Since this is a long article, I recommend reading it in your browser, where you can access the table of contents menu on the left side.
为求完整，在把 Jev 放入上下文（此处并无双关之意）之前，我认为按时间顺序展开最为合理。在本节中，我想简要梳理一下在基于 Transformer 的模型出现之前，通过朴素贝叶斯、逻辑回归以及更经典的（深度）神经网络所进行的应用文本分类。
For completeness, before we put Jev in context (no pun intended), I thought it made the most sense to start chronologically. In this section, I want to take a brief tour of applied text classification via naive Bayes, logistic regression, and the more classic (deep) neural networks before transformer-based models came along.
回想当年，15 年前我还是研究生的时候，尽管循环神经网络已经存在（后文会详细讨论），文本分类通常仍采用词袋（bag-of-words）表示，因为它简单直接，并且能在中等规模数据集上取得不错的效果。
Back in the day, when I was a grad student 15 years ago, even though recurrent neural networks already existed (more on that later), text classification was usually done with a bag-of-words representation because it was straightforward and could get good results on moderately sized datasets.
简而言之，我们可以把词袋表示看作一种方法，它使长度不一的自由形式文本输入能够与经典分类器（朴素贝叶斯、逻辑回归、SVM、随机森林、XGBoost 等）兼容，而这些分类器期望的是固定大小的输入向量。
In short, we can think of the bag-of-words representation as a method that makes free-form text input of different lengths compatible with classic classifiers (naive Bayes, Logistic Regression, SVMs, Random Forest, XGBoost, to name a few), which expect a fixed-size input vector.
流行的实际应用包括从新闻文章分类到电子邮件垃圾邮件过滤等各种场景。而且，据说就连 Gmail 最初的垃圾邮件过滤器也使用了基于词袋表示的朴素贝叶斯模型。
Popular real-world applications include anything from news article classification to email spam filtering. And yes, allegedly even Gmail's original spam filter used a Naive Bayes model with a bag-of-words representation.
顺便提一句，我正好在 12 年前写过关于这种方法的文章。那是我在 arXiv 上分享的最早的内容之一。
As a side note, I wrote about this approach exactly 12 years ago. It was one of the first things I shared on arXiv.
图 2：我 2014 年的一份旧教程，用词袋模型讲解朴素贝叶斯分类器。
Figure 2: An old tutorial of mine from 2014 that explains naive Bayes classifiers using a bag-of-words model.
使用朴素贝叶斯的文本分类至少可以追溯到 1961 年，当时它被用于对计算机摘要进行分类；参见《自动索引：实验探究》一文。
Text classification with Naive Bayes goes back to at least 1961, where it was used to sort computer abstracts; see the Automatic Indexing: And Experimental Inquiry paper.
那么，究竟什么是词袋表示？它是一种将不同长度的自由形式文本转换为固定长度表示的方法，例如：
So, what exactly is this bag-of-words representation? It’s a way to convert free-form texts with different lengths, e.g.,
* 训练示例 1：_“《Zentropa》是我多年来看过的最具原创性的电影。如果你喜欢受黑色电影影响的独特惊悚片，那么这正是治疗所有那些如今堵塞影院的夏季好莱坞大片的良药。冯·提尔的后续作品如《破浪》获得了更多赞誉，但这确实是他最好的作品。”_
* Training example 1: _“Zentropa is the most original movie I’ve seen in years. If you like unique thrillers that are influenced by film noir, then this is just the right cure for all of those Hollywood summer blockbusters clogging the theaters these days. Von Trier’s follow-ups like Breaking the Waves have gotten more acclaim, but this is really his best work.”_
* 训练示例 2：_“这部电影简直太糟糕了。约翰·里特表演笨拙的摔倒，75%的演员念台词就像在念提词卡，剪辑糟糕，混音恐怖。”_
* Training example 2: _“This film is just plain horrible. John Ritter doing pratt falls, 75% of the actors delivering their lines as if they were reading them from cue cards, poor editing, horrible sound mixing”_
* 训练示例 3：_“《Zentropa》与《第三人》有很多共同之处，这是另一部类似黑色电影的作品，背景设定在战后欧洲的废墟中。”_
* Training example 3: _“Zentropa has much in common with The Third Man, another noir-like film set among the rubble of postwar Europe.”_
转换为固定大小的表示，以供上述“经典”分类器使用。（上例摘自常用的 IMDb 电影评论分类数据集。）
into a fixed-size representation for the aforementioned "classic" classifiers. (The example above is an excerpt from the popular IMDb movie review classification dataset.)
图 3：词袋表示示意图。
Figure 3: An illustration of a bag-of-words representation.
词袋模型首先构建词表，词表由训练集中所有不重复的词组成（可选地，可以去除所谓的停用词，如“a”和“the”，这些词在大多数语境中几乎不携带语义信息）。
A bag-of-words model starts by building the vocabulary, which consists of all unique words in the training set (optionally, one can get rid of so-called stopwords like "a" and "the", which are words that carry little to no semantic meaning in most contexts).
词袋表示通过为词表中的每个词在向量中分配一个位置，从而得到这些固定大小的输入。然后我们统计每个词在文档中出现的次数。例如，若词表包含 50,000 个不重复的词，则无论输入只有十个词还是 30 万个词，都会生成一个包含 50,000 个元素的固定大小向量。注意，大多数元素为零，因为每个文档只包含词表的一小部分。（除了表示原始计数外，还有 TF-IDF 等归一化方案。）
A bag-of-words representation results in these fixed-size inputs by assigning each word in a vocabulary its own position in a vector. We then count how often each word occurs in a document. For example, if we have a vocabulary of 50,000 unique words, it produces a fixed-size vector with 50,000 entries, regardless of whether the input consists of only ten words or 300k words. Note that most entries are zero because each document contains only a small subset of the vocabulary. (Instead of representing the raw counts, there are also normalization schemes like TF-IDF.)
然后，一旦得到这些词频向量，我们就可以在带标签的训练集上训练分类器，例如将邮件标注为垃圾邮件或非垃圾邮件。例如，逻辑回归模型随后会学习特征权重，使某些词（及其词频）与特定标签相关联。例如，某些词可能提高预测的垃圾邮件概率，而另一些词则可能降低该概率。
Then, once we have these word frequency vectors, we can train a classifier on a labeled training set, such as emails labeled as spam or non-spam. For example, a logistic regression model would then learn feature weights that correlate certain words (and word counts) with particular labels. For instance, certain words might increase the predicted spam probability, and others may decrease it.
这种方法计算成本低，并且当特定词语为标签提供强烈线索时，效果很好。在简单的分类任务（如垃圾邮件分类）中，这通常足以快速获得合理准确的结果。
This approach is computationally cheap and can work well when particular words provide strong clues about the label. In a simple classification task such as spam classification, this is often enough to get quick, reasonably accurate results.
但这种方法最大的缺点之一是，由于词袋表示的本质，它丢失了词序。因此，例如，“狗咬人”和“人咬狗”尽管描述不同事件，却产生相同的向量。
But one of the biggest downsides of this approach is that, because of the nature of the bag-of-words representation, it loses word order. So, for example, “the dog bites the man” and “the man bites the dog” produce identical vectors despite describing different events.
（有一些变通方法可以通过添加词对或更长的序列（称为 n-gram）作为特征来保留一些局部顺序，但这会增加词汇量。）
(There are some workarounds to preserve some local order by adding word pairs or longer sequences, called n-grams, as features, although this increases the vocabulary size.)
尽管有这些缺点，我仍然认为词袋在某些低风险应用中有其用武之地，因为它非常便宜，而且词袋表示 + 逻辑回归仍然是我处理每个文本分类问题的首选基线，因为它非常容易实现。
Despite the shortcomings, I still think that a bag-of-words has its place in certain low-stakes applications because it’s so cheap, and a bag-of-words representation + logistic regression remains my go-to baseline for every text classification problem, since it’s so easy to implement.
图 4：对于那些有兴趣构建简单逻辑回归分类器的人，我在这里有一个教程。该模型达到了 89.9% 的准确率（在平衡数据集上）。
Figure 4: For those interested in building a simple logistic regression classifier, I have a tutorial up here. This model achieves 89.9% accuracy (on a balanced dataset).
上述词袋模型同样可以配合（简单的）深度神经网络使用，例如多层感知机。但其缺点依然存在：我们会丢失句子结构和词序信息。
The aforementioned bag-of-words model would also work with (simple) deep neural networks, like multilayer perceptrons. But the downside still is that we would lose the sentence structure and word order.
然而，更复杂的神经网络架构可以避开词袋这一权宜之计：卷积神经网络（CNN）和循环神经网络（RNN），它们可以将词嵌入作为输入。
However, more sophisticated neural network architectures avoid the bag-of-words workaround: convolutional neural networks (CNNs) and recurrent neural networks (RNNs), which can take word embeddings as input.
首先，在将输入文本送入模型之前，我们必须将其转换为合适的表示形式。一种这样的表示是词袋（bag-of-words）。另一种是词嵌入向量。二者的区别在于：词袋向量通过统计每个词汇在文本中出现的次数来表示整个文本，而词嵌入则将单个词表示为一个由学习得到的数值构成的稠密向量。
First, before feeding the input texts into a model, we have to convert them into a suitable representation. One such representation is bag-of-words. Another is word embedding vectors. The difference is that a bag-of-words vector represents the entire text by counting how often each vocabulary word occurs, while a word embedding represents an individual word as a dense vector of learned numbers.
图 5：创建词嵌入的示意图。
Figure 5: Illustration of creating word embeddings.
* 词嵌入的工作原理类似于 LLM 中的嵌入层，即它们将输入 token 转换为稠密向量。嵌入可以在模型之外进行（例如，两种经典且流行的学习方法是 Word2Vec 和 GloVe），嵌入层也可以是神经网络架构本身的一部分，并在模型训练过程中被学习和调整。
* Word embeddings work similarly to embedding layers in LLMs, i.e., they convert input tokens into dense vectors. Embeddings can happen outside the model (e.g., two classic, popular methods for learning them are Word2Vec and GloVe), or the embedding layer can be part of the neural network architecture itself and be learned and tuned during model training.
这些经典嵌入在查表时是与上下文无关的。例如，单词“bank”在“river bank”和“bank account”中得到相同的向量（如你所知，上下文可以通过注意力机制等概念来处理）。
These classic embeddings are context-independent at lookup time. The word "bank", for example, gets the same vector in "river bank" and "bank account" (as you may know, context can be handled via concepts like attention).
关于嵌入的更多资源，以下内容可能对你有所帮助：
For additional resources on embeddings, you may find the following ones helpful:
* 第 2 章：处理文本数据（本章属于 LLM 内容，但应能让你大致理解如何嵌入词或 token；在 LLM 分词器中，我们会把词切分为子词 token；而在 Word2Vec 中，1 个词通常就是 1 个 token。）
* Chapter 2: Working with Text Data (this is an LLM chapter but should give you the gist of embedding words or tokens; in LLM tokenizers, we split words into subword tokens; in Word2Vec, 1 word is usually 1 token.)
* 理解嵌入层与线性层的区别（此处说明嵌入向量在数学上何时等价于线性层与矩阵乘法。）
* Understanding the Difference Between Embedding Layers and Linear Layers (this is an illustration that shows when embedding vectors are mathematically equivalent to Linear layers and matrix multiplications.)
由于许多人可能已经熟悉循环神经网络（RNN），我将简短介绍本节。RNN 是自然语言处理中经典的神经网络架构，其流行变体可追溯到 20 世纪 80 年代和 90 年代初。2017 年引入的 Transformer（使用了最初在 RNN 中引入的注意力机制；简要时间线见我的《理解大型语言模型》）随后在许多 NLP 应用中逐渐取代了它们。
Since many of you are probably familiar with recurrent neural networks (RNNs), I will keep this section short. RNNs are a classic go-to neural network architecture for natural language processing, and popular variants go back to the 1980s and early 1990s. Transformers, which were introduced in 2017 (and using the attention mechanisms that were first introduced in RNNs; see my Understanding Large Language Models for a brief timeline), then gradually replaced them in many NLP applications.
RNN 一次读取一个词组成的序列（如文本）。每一步，它将当前词嵌入（上一节讨论）与上一步的隐藏状态结合。我们可以将隐藏状态视为一个固定大小的向量，它概括了迄今为止处理的文本，因此这使得词序变得重要，因为重新排列词语会改变状态更新的序列。
RNNs read a sequence (like text) one word at a time. At each step, they combine the current word embedding (discussed in the previous section) with a hidden state from the previous step. We can think of the hidden state as a fixed-size vector that summarizes the text processed so far, so this makes word order matter, because rearranging the words changes the sequence of state updates.
图 6：RNN 分类器的示意图。
Figure 6: Illustration of an RNN classifier.
注意，上图展示了展开表示中的 RNN。即，RNN 对每个输入重用相同的层堆栈，因此称为“循环”。并且由于是“循环”的，输入文本可以具有任意长度。下图并排展示了展开表示中的“循环”。注意两者展示的是相同的架构，只是可视化方式不同。
Note that the figure above shows the RNN in the unrolled representation. I.e., the RNN reuses the same layer stack for each input, hence the term “recurrent”. And since it’s “recurrent”, the input text can have an arbitrary length. The figure below illustrates the “recurrence” with the unrolled representation side by side. Note that both show the identical architecture, it’s just a different visualization.
图 7：同一 RNN 的折叠和展开示意图。
Figure 7: Rolled and unrolled illustration of the same RNN.
RNN 众所周知难以训练，并且 RNN 有一些重要的改进，例如 1997 年提出的长短期记忆（LSTM）网络和 2014 年提出的门控循环单元（GRU），它们使用学习到的门来控制信息的保留和更新方式。（还有更近期的 xLSTM：扩展长短期记忆，于 2024 年提出）。
RNNs were notoriously hard to train, and there are important improvements to RNNs, like Long short-term memory (LSTM) networks, introduced in 1997, and gated recurrent units (GRUs), introduced in 2014, which use learned gates to control how information is retained and updated. (There is also the more recent xLSTM: Extended long short-term memory, introduced in 2024).
此外，状态空间模型受到这种固定大小隐藏状态顺序更新的思想的启发，这比 Transformer 的注意力机制成本更低。然而，瓶颈仍然是隐藏状态能够保留多少信息，并且它仍然必须顺序处理。（有趣的事实：注意力机制最初是为 RNN 开发的，之后 Transformer 架构才出现，但那是另一个故事了；我在我的《理解大型语言模型》文章中写过这一点。）
Also, state-space models are inspired by this idea of a fixed-size hidden state updated sequentially, which is cheaper than transformer attention. However, the bottleneck is still how much information the hidden state can retain, and it still has to be processed sequentially. (Fun fact: attention was first developed for RNNs before the transformer architecture came along, but it’s a story for another time; I’ve written about it in my Understanding Large Language Models article.)
底线是 RNN 可以用于训练文本分类器。回到 IMDb 电影评论数据集，使用逻辑回归的词袋分类器达到了约 89.9% 的准确率（在平衡数据集上），而 LSTM RNN 仅达到 85.66% 的准确率。是的，RNN 可能更难训练（请继续关注下面的 ULMFiT 方法，它训练的 RNN 准确率要高得多）。
The bottom line is that RNNs can be used to train text classifiers. Coming back to the IMDb movie review dataset, the bag-of-words classifier with logistic regression achieved about 89.9% accuracy (on a balanced dataset), while an LSTM RNN achieved only 85.66% accuracy. Yes, RNNs can be harder to train (stay tuned for the ULMFiT method below, which trains an RNN with much higher accuracy).
图 8：一个简单的 RNN 与 LSTM 教程。该模型达到了 85.66% 的准确率（在平衡数据集上）；较高的训练准确率表明存在严重的过拟合。
Figure 8: A simple RNN with LSTM tutorial. This model achieves 85.66% accuracy (on a balanced dataset); the higher training accuracy indicates substantial overfitting.
请注意，这个 RNN 是从头开始训练的。更好的方法是先在更大的数据集上预训练模型，然后在这个目标数据集上微调（经典地，我们称这种方法为“迁移学习”）。
Note that this RNN was trained from scratch. A better approach is to pre-train the model on a larger dataset first, then fine-tune it on this target dataset (classically, we call this approach “transfer learning”).
在自然语言处理领域，提出该方法的最具影响力的论文之一是 ULMFiT（2018），它在 IMDb 上取得了令人印象深刻的 95.4% 测试准确率。
In the natural language processing domain, one of the most influential papers proposing this approach is ULMFiT (2018), which achieved an impressive 95.4% test accuracy on IMDb.
图 9：ULMFiT 论文中的标注图。
Figure 9: Annotated figure from the ULMFiT paper.
你大概是通过卷积神经网络（CNN）在计算机视觉中的应用而认识它的。然而，将其用于文本也是可行的，尽管历史上并不那么常见。
You probably know convolutional neural networks (CNNs) from their use in computer vision. However, it is also possible, although historically less common, to use them for text.
图 10：用于图像分类的卷积神经网络（CNN）示意图。
Figure 10: Illustration of a convolutional neural network (CNN) to classify images.
如上图图像分类示例所示，CNN 将学习到的滤波器应用于图像块（窗口）。类似地，在自然语言领域，我们可以将学习到的滤波器应用于相邻词嵌入的窗口。
As shown in the image classification example in the figure above, CNNs apply learned filters to image patches (windows). Similarly, in the natural language domain, we can apply learned filters to windows of adjacent word embeddings.
图 11：用于文本分类的 CNN 分步示意。为简单起见，仅展示 1 个滤波器（通道）。
Figure 11: A CNN for text classification, step by step. Only 1 filter (channel) for simplicity.
如上图所示，窗口大小为 3 的卷积滤波器对每三个相邻词组成的组使用相同的权重。随后，当它在输入上滑动时，会一次移动一个词，扫过“the movie had surprisingly good acting”。因此，这会产生四个窗口（为简单起见忽略填充）：
As illustrated in the figure above, a convolutional filter with a window size of three uses the same weights for each adjacent group of three words. Then, as it slides over the inputs, it moves the filter one word at a time across "the movie had surprisingly good acting". So, this gives four windows (ignoring padding for simplicity):
因此，在最后一层、分类头之前，我们可以先对结果做展平（flatten）或全局最大池化（global max pooling），再将其连接到分类头。展平虽然保留了全部信息，但会产生长度随文本输入长度而变化的向量。（例如，如果“the movie had surprisingly good acting”更长，我们就会得到更长的特征图。）因此，为了使模型对输入长度不敏感（input-length agnostic），全局最大池化在这里会是更好的选择。
So, in the last layer, before the classification head, we can either flatten or global max pool the results before connecting it to the classification head. While flattening preserves all information, it would produce differently sized vectors depending on the text input length. (E.g., if "the movie had surprisingly good acting" were longer, we would have longer feature maps.) So, to make it input-length agnostic, global max-pooling would be a better option here.
简而言之，我们可以将文本 CNN 可视化如下。
In short, we can visualize the text CNN as shown below.
图 12：使用多个滤波器（通道）进行文本分类的 CNN。
Figure 12: CNN for text classification with multiple filters (channels).
此外，一个优点在于，各位置上的滤波器计算可以并行执行，从而避免了 RNN 逐步依赖的问题。
Also, what's nice is that the filter computations across positions can run in parallel, which avoids the step-by-step dependency of an RNN.
为便于快速比较，在上述 IMDb 数据集上，我的实验表明，这样的 CNN 可获得约 90.07% 的准确率（但请注意，这高度依赖于架构；例如，你可能从计算机视觉领域了解到，准确率可能差异很大）。（例如，经典的 AlexNet 在 ImageNet 上的 top-1 准确率约为 62.5%，而 ConvNeXt V2-H 则达到 88.9%。）
For a quick comparison, on the aforementioned IMDb dataset, my experiments show that such a CNN gets about 90.07% accuracy (but note that this is highly architecture-dependent; for example, you may know from computer vision contexts that accuracies can vary widely). (E.g., the good old AlexNet had a ~62.5% top-1 accuracy on ImageNet, and a ConvNeXt V2-H gets 88.9%.)
图 13：一个经训练用于对 IMDb 电影评论进行分类的 CNN；它在该平衡数据集上达到 90.07% 的准确率。源代码可在此处获取。
Figure 13: A CNN trained to classify IMDb movie reviews; it achieves 90.07% accuracy on this balanced dataset. You can find the source code here.
2017 年，原始的 Transformer 架构在《Attention Is All You Need》一文中被提出。由于这一主题已被（我和其他人）如此广泛地讨论过，我将聚焦于与分类相关的方面。但对于那些对注意力机制及其他架构细节感兴趣的读者，请参阅我的相关文章：
In 2017, the original transformer architecture was introduced in the *Attention Is All You Need* paper. Since this topic has been covered so extensively (by me and others), I will focus on the classification-relevant aspects. But for those interested in the attention mechanism and other architecture details, please see my related articles:
#### 大型 LLM 架构比较 [Sebastian Raschka 博士 · 2025 年 7 月 19 日 阅读全文](https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison)
#### The Big LLM Architecture Comparison [Sebastian Raschka, PhD · July 19, 2025 Read full story](https://magazine.sebastianraschka.com/p/the-big-llm-architecture-comparison)
#### 现代 LLM 中注意力机制变体的可视化指南 [Sebastian Raschka 博士 · 3 月 22 日 阅读全文](https://magazine.sebastianraschka.com/p/visual-attention-variants)
#### A Visual Guide to Attention Variants in Modern LLMs [Sebastian Raschka, PhD · Mar 22 Read full story](https://magazine.sebastianraschka.com/p/visual-attention-variants)
关键在于，原始的 Transformer 架构是一种用于语言翻译的编码器-解码器结构，但它可以很容易地适配文本分类任务，正如我将在后续章节中说明的那样。
The key point is that the original transformer architecture was an encoder-decoder setup used for language translation, but it can be easily adapted for text classification tasks, as I'll illustrate in the following sections.
图 14：来自《Attention Is All You Need》的原始 Transformer 架构。
Figure 14: The original transformer architecture from *Attention Is All You Need*.
我至今仍清楚地记得原始 Transformer 架构发布后的最初几年。那几年基本上是由两种不同路线之间的竞争所定义的：
I remember all too well the first years after the original Transformer architecture release. They were pretty much defined by the rivalry between two different approaches:
1. 以 BERT 为代表的编码器式模型，主要由 Google 开发；
1. Encoder-style models like BERT, primarily developed by Google;
2. 以 GPT 为代表的解码器式模型，主要由 OpenAI 开发。
2. Decoder-style models like GPT, primarily developed by OpenAI.
编码器式模型天生就是文本分类器，而 GPT 模型能够以涌现特性的形式完成零样本和少样本分类，但其优势更多体现在生成任务上。
Encoder-style models were natural text classifiers, whereas GPT models could do zero- and few-shot classification as an emergent property, but their strength was more in generative tasks.
不过，让我们先从编码器式模型讲起，看看如何对它们进行微调并用它们来对文本进行分类。
But let's start with encoder-style models and how to fine-tune and use them to classify text.
使用 Transformer（无论是编码器式还是解码器式）的一个普遍共性是：我们所用的是在大规模文本语料上预训练的模型，并且除了将其用作零样本或少样本分类器之外，我们还可以在目标数据集上对其进行微调（类似于前文 RNN 章节中提到的 ULMFiT）。
One universal aspect of using transformers, whether encoder- or decoder-style, is that we work with models pre-trained on large text corpora, and, in addition to using them as zero- or few-shot classifiers, we can fine-tune them on the target dataset (similar to ULMFiT, as mentioned in the RNN section earlier).
如下方摘自 BERT 论文（2018）的图所示，BERT/编码器式模型在第一个位置设有一个分类 token，我们可以方便地针对该 token 进行微调。
As shown in the figure excerpt from the BERT paper (2018) below, BERT-/encoder-style models have a classification token at the first position that we can conveniently fine-tune for this.
图 15：原始 BERT 论文中的标注图。
Figure 15: Annotated figure from the original BERT paper.
尽管 BERT 模型远不如自回归 GPT 式 Transformer 那样流行，但所幸仍有人偶尔对它们进行更新与现代化改造。近期的一个例子是 2024 年的 ModernBERT 模型，它常是我处理分类任务时的首选。
While BERT models are not nearly as popular as autoregressive GPT-style transformers, thankfully some people still update and modernize them occasionally. One recent example that is often my go-to for classification tasks is the 2024 ModernBERT model.
如下所示，在 IMDb 电影评论数据集上，ModernBERT 只需极少的微调工作即可达到约 95% 的准确率。
As shown below, on the IMDb movie reviews, ModernBERT gets approximately 95% accuracy with very little fine-tuning effort.
图 16：不同 LLM 经微调后用于对 IMDb 电影评论进行分类；源代码可在此处找到。（我做的超参数调优非常少，这一结果还有望再提升 1–2%；不过，这里想说明的一点也是：只需极少的投入就能取得相当不错的结果。）
Figure 16: Different LLMs fine-tuned to classify IMDb movie reviews; you can find the source code here. (I did very little hyperparameter tuning, and this can potentially be improved by 1-2% further; however, one of the points here is also that one can get really good results with pretty minimal effort.)
像 GPT 这样的 LLM 是解码器式、自回归的 Transformer，它们是生成文本和代码时所有注意力机制的核心（并非双关）。
LLMs like GPT are decoder-style, autoregressive transformers that are the center of all attention (no pun intended) for generating text and code.
然而，正如我在《Build A Large Language Model（From Scratch）》一书第 6 章中所解释的，作为微调的温和入门（在介绍指令微调之前），我们也可以将这些模型重新用于文本分类。
However, as I explained in chapter 6 of my Build A Large Language Model (From Scratch) book, as a gentle introduction to fine-tuning (before covering instruction fine-tuning), we can also repurpose these for text classification.
当然，我们也可以直接对 LLM 进行提示，如下方截图所示。
Sure, we can also prompt an LLM directly, as shown in the screenshot below.
图 17：提示 LLM 对一条电影评论进行分类。
Figure 17: Prompting an LLM to classify a movie review.
然而，如果我们想要结构化输出，并且心中已有特定的目标领域，这种做法就过于脆弱且低效。
However, if we want structured outputs and we have a specific target domain in mind, this is unnecessarily brittle and inefficient.
相反，我们可以将输出层替换为一个更精简的分类头，如下图所示：
Instead, we can replace the output layer with a leaner classification head, as illustrated below:
图 18：将 GPT 式模型的输出层替换为更精简的分类头。
Figure 18: Swapping the output layer of a GPT-style model with a leaner classification head.
不过，微调存在一些注意事项。例如，由于自回归注意力掩码的存在，我们在设计微调时必须格外小心，以确保分类 token 能够获取序列中所有其他 token 的信息。
Now, fine-tuning has a few caveats. For instance, because of the autoregressive attention mask, we have to be careful about how we design fine-tuning so that the classification token has information about all other tokens in the sequence.
图 19：BERT 与 GPT 中的注意力掩码。
Figure 19: Attention masks in BERT and GPT.
如果您对更多技术细节感兴趣，请参阅我最近那篇端到端的《从零构建 AI 文本检测器》文章：
If you are interested in further technical details, please see my recent end-to-end Building an AI Text Detector From Scratch article:
尽管最初的 Transformer 架构（一种编码器-解码器结构）被拆分为两种范式——如 BERT 的编码器式模型和如 GPT 的解码器式 LLM，但也有人尝试使用编码器-解码器式的变体，其中最新的变体（多少有些出人意料）是 DeepSeek V4.1 Flash。（不过，在这种情况下，它是因果编码器，而非 T5 中那样的双向编码器。）
While the original transformer architecture (an encoder-decoder) was split into two paradigms, encoder-style models like BERT and decoder-style LLMs like GPT, there were also efforts to use encoder-decoder-style variants, with the most recent variant (somewhat unexpectedly) being DeepSeek V4.1 Flash. (However, in this case, it's a causal encoder not a bidirectional one as in T5.)
将焦点保持在用于分类的编码器-解码器架构上，最突出的候选者大概是 Google 2019 年的 T5（Text-to-Text Transfer Transformer）。
Keeping the focus on encoder-decoder architectures for classification, probably the most prominent candidate is Google's 2019 T5 (Text-to-Text Transfer Transformer).
在架构上，T5 与原始 Transformer 的差异包括对架构的更新（总结如下），以及对训练方式的改变。原始 Transformer 被训练用于（以监督方式）进行语言翻译；T5 则使用带有 span corruption 的未标注文本预训练，其中编码器接收缺失 span 的文本，解码器生成这些缺失的 span。
Architecturally, the differences between T5 and the original transformer include updates to the architecture (as summarized below), as well as changes to training. The original transformer was trained to do language translation (in a supervised fashion); T5 uses pre-training on unlabeled text with span corruption, where the encoder receives text with missing spans and the decoder generates those missing spans.
图 21：T5 对原始 Transformer 架构的架构改动。（架构图取自 Attention Is All You Need。）
Figure 21: T5 architecture changes to the original transformer architecture. (The architecture drawing is taken from Attention Is All You Need.)
现在，我们可以像上面其他 RNN、CNN 和 Transformer 方法一样，通过添加一个分类头来使用 T5。
Now, we can use T5 similar to the other RNN, CNN, and transformer approaches above, by adding a classification head.
此外，我们也可以（通过训练）让解码器输出类别标签预测（例如 IMDb 电影评论数据集中的“正面”或“负面”），这与常规 LLM 类似。我们将这种方法称为文本到文本分类。
Additionally, we can also (train to) have the decoder output the class label prediction (like "positive" or "negative" in the IMDb movie review dataset case), similar to regular LLMs. Let's call this approach text-to-text classification.
尽管 GPT 式 LLM 是在海量文本上训练的，但它们通常开箱即能很好地完成文本到文本分类，如前所示（为方便起见，下图再次插入于此）。
While GPT-style LLMs are trained on massive amounts of text, they usually perform text-to-text classification well out of the box, as shown earlier (the figure below is inserted here again for convenience).
图 22：使用 GPT 模型进行文本到文本分类。
Figure 22: Text-to-text classification with a GPT model.
然而，在 T5 的情况下，通常会进一步微调解码器，使其在给定目标领域的这类任务上表现良好。
However, in the case of T5, it is common to further fine-tune the decoder to do well on these types of tasks in a given target domain.
对于 T5，两种方法都可行。总结如下：
For T5, both approaches work. Here's a summary:
图 23：分类头与文本到文本方法的对比。
Figure 23: Classification head vs text-to-text approaches.
到目前为止，我们已经看到文本分类有大量方法，从使用词袋表示的逻辑回归和朴素贝叶斯等传统方法，到提示最新的前沿 LLM（如 GPT-6），或使用分类头（或文本到文本分类）微调任何开放权重 LLM。
So far, we have seen that there are plenty of approaches to text classification, from traditional methods like logistic regression and naive Bayes with bag-of-words representations to prompting the latest frontier LLMs like GPT-6 or fine-tuning any open-weight LLM with a classification head (or text-to-text classification).
起初，我（几乎）把它当作“只是一个分类器”而不予重视——这类东西我经常为自然语言输入的分类任务而构建（例如，参见我的 AI Detector 文章，作为最近的一个公开示例）。
At first, I (almost) dismissed it as "just a classifier," something that I build routinely for classification tasks with natural language inputs (e.g., see my AI Detector article as a recent public example).
不过，在继续讨论之前，让我们先快速概述一下 Jev。Jev 是 TypeSafe AI 发布的一个新模型，几周前刚刚结束隐身模式，引起了大量关注（起初，它看起来有点奇怪，因为它看起来就像一个分类器）。
Before continuing this discussion, though, let's start with a quick Jev overview. Jev is a new model released by TypeSafe AI, which just came out of stealth a few weeks ago and got a lot of attention (at first, it seemed a bit bizarre because it looks just like a classifier).
Jev 是一个专有模型（尽管发布后出现了大量快速开源克隆，但我们稍后会讨论这一点），使用成本相对较低，并声称在决策方面与 GPT-5.6 Luna 相当，同时速度快几个数量级且成本更低：
Jev is a proprietary model (although the release was followed by a huge number of quick open-source clones, but we will get to this later) that is relatively cheap to use and claims to be on par with GPT-5.6 Luna (for decision-making) while being orders of magnitude faster and cheaper:
图 24：来自 TypeSafe AI 博客的基准测试。
Figure 24: Benchmark from TypeSafe AI blog.
这里，关于 GPT-Luna，可以将其视为前面讨论的文本到文本分类方法。
Here, regarding GPT-Luna, think of this as the text-to-text classification approach discussed earlier.
那么为什么会有这么多炒作呢？我认为部分原因是其友好的 API 以及在各种任务上表现优异，因此不需要自定义微调。
So why all this hype? I think it's partly because of the nice API and that it performs so well on all kinds of tasks, so it doesn't require custom fine-tuning.
例如，我可以用它来对支持工单/邮件进行分类（我制作了一个视频版本来展示响应时间）：
For example, I can use it to categorize support tickets/emails (I made a video version to show the response time):
或者，我可以让同一个模型实时玩俄罗斯方块（这里我使用的是 Choice API）：
Or, I can have the same model play Tetris in real-time (here, I am using the Choice API):
也许最简洁地解释它的方式（在深入技术细节之前）如下：就像 2022 年的 ChatGPT 之所以令人兴奋，是因为它是一个通用聊天模型，能够生成各种文本；技术社区对 Jev 感到兴奋的原因之一在于，它是分类领域的 ChatGPT 时刻——它可以低成本地对各种文本输入进行分类，而无需为每个任务微调一个自定义分类器。
Maybe the best way to succinctly explain it (before getting into technical details) is as follows: Like ChatGPT in 2022 was exciting because it was a general-purpose chat model that could generate all kinds of texts, one of the reasons the tech community is excited about Jev is that it is the ChatGPT moment for classification, where it can cheaply classify all kinds of text inputs without having to fine-tune a custom classifier for each task.
（截至本文写作时，关于其架构和确切训练算法所知甚少，只知道其中一位创始人说它是通过“用于校准决策的强化学习”训练的，但这一点稍后再详述。）
(As of this writing, there's not much known about the architecture and exact training algorithm, except that one of the founders said it was trained via "Reinforcement Learning for Calibrated Decisions", but more on that later.)
在过去几年里，我们已经习惯于用更大、更好、更昂贵的 GPT 风格 LLM 来解决各种问题，而对于有针对性的决策或分类任务，像 Jev 这样廉价且快速的方法可能会让大多数人感到耳目一新。特别是对于一次性任务，收集训练数据并微调自定义 ModernBERT 听起来太繁琐，我们可能只需扔一个类似 Luna 的模型给它。
Over the past few years, we’ve gotten used to throwing bigger, better, and more expensive GPT-style LLMs at all kinds of problems, and for targeted decision-making or classification tasks, a cheap & fast approach like Jev may feel refreshing to most. Especially for one-off tasks where collecting training data and fine-tuning a custom ModernBERT sounds too tedious, we might just throw a Luna-like model at it.
但除了因其多功能性（即在不同目标领域开箱即用表现良好）而受欢迎之外，Jev 还有一个相对不错的 API，我们可以通过终端中的 curl 或通过其 Python API 使用。
But on top of being popular for its versatility (that is, performing well on different target domains out of the box), Jev also has a relatively nice API which we can use via curl in the terminal or via its Python API.
简而言之，下面展示了三种主要的 API 类型。让我们从 Choice API 开始，它便于多类分类。
In short, there are three main API types illustrated below. Let’s start with the Choice API, which is convenient for multi-class classification.
图 25：Jev 的 Choice API。用于多类分类。
Figure 25: Jev’s Choice API. Use this for multi-class classification.
接下来是 Noul API，它更简单，并为问题分配一个“是”的概率。
Next is the Noul API, which is simpler and assigns a “yes” probability to a question.
图 26：Jev 的 Noul API。用于二分类或多标签分类（配合多个 Noul 问题）。
Figure 26: Jev's Noul API. Use this for binary classification or multi-label classification (with multiple Noul questions).
最后，Score API 根据评分量表等级（下例中为 0、1、2）给出分数。
Lastly, the Score API assigns a score based on a rubric level (in the example below, 0, 1, 2).
图 27：Jev 的 Score API。用于有序分类。
Figure 27: Jev's Score API. Use this for ordinal classification.
总结起来，不同 API 及其用例分别如下。
To sum it up, the different APIs and use cases are as follows.
为了让这些 API 更加具体，并回答 Jev 在上述 IMDb 数据集上可能表现如何这一问题，我们可以用 Choice 或 Noul API 来运行它。
To make these APIs more concrete, and to answer the question of how well Jev might do on the aforementioned IMDb dataset, we can run it with either the Choice or Noul API.
我们先从 Choice API 开始。每条评论的格式如下：
Let's start with the Choice API first. Each review would be formatted as follows:
curl -sS https://api.typesafe.ai/v1/systemone \
curl -sS https://api.typesafe.ai/v1/systemone \
-H "Authorization: Bearer $TYPESAFE_API_KEY" \
-H "Authorization: Bearer $TYPESAFE_API_KEY" \
"state": "The acting was excellent and the story kept me engaged throughout. I would happily watch this movie again.",
"state": "The acting was excellent and the story kept me engaged throughout. I would happily watch this movie again.",
"instructions": “这条电影评论的整体情感是什么？”,
"instructions": "What is the overall sentiment of this movie review?",
"negative": “对这部电影整体上持负面看法”,
"negative": "An overall unfavorable opinion of the movie",
"positive": “对这部电影整体上持正面看法”
"positive": "An overall favorable opinion of the movie"
如果我们想为二分类（此处）或多分类问题使用显式标签，这个 Choice API 就很方便。
This Choice API is convenient if we want to use explicit labels for binary (here) or multi-class problems.
实际响应可能如下所示：
An actual response might look as follows:
除了“choice”：“positive”标签外，我们还获得了该预测的置信度，以及每个类别的概率。置信度在实际应用中非常有用。（置信度字段概括了概率分布的集中程度，它与分配给获胜类别的概率不同。）根据文档，Jev 的卖点之一是这些概率是良好校准的（但校准的更多内容将在后面讨论）。
Along with the "choice": "positive" label, we also get the confidence for this prediction, which can be really useful in real-world applications, as well as the probabilities for each class. (The confidence field summarizes how concentrated the probability distribution is. It is different from the probability assigned to the winning class.) One of the selling points of Jev is that, according to the documentation, these are well-calibrated (but more on calibration later).
或者，我们也可以在此使用 Noul API 对评论进行分类。使用相同的电影评论分类上下文，格式如下：
Alternatively, we can also use the Noul API here to classify the reviews. Using the same movie review classification context, the format would be as follows:
curl -sS https://api.typesafe.ai/v1/systemone \
curl -sS https://api.typesafe.ai/v1/systemone \
-H "Authorization: Bearer $TYPESAFE_API_KEY" \
-H "Authorization: Bearer $TYPESAFE_API_KEY" \
"state": "The acting was excellent and the story kept me engaged throughout. I would happily watch this movie again.",
"state": "The acting was excellent and the story kept me engaged throughout. I would happily watch this movie again.",
"instructions": “这条评论是否表达了对电影的总体正面评价？”,
"instructions": "Does this review express an overall positive opinion of the movie?",
"true": “对电影的总体正面评价”,
"true": "An overall favorable opinion of the movie",
"false": “对电影的总体负面评价”
"false": "An overall unfavorable opinion of the movie"
（有趣的是，尽管文本相同，它对正类给出的概率是 0.98 而非 1.0。）
(Interestingly, it gives 0.98 instead of 1.0 for the positive class, even though it's the same text.)
我们还可以通过为每个类别各提一个问题，将 Noul API 应用于多类别分类，例如：
We can also apply the Noul API to multiple classes by asking one question per class, for example:
在这里，每个问题（彼此独立）返回一个概率，这些概率不必总和为 1。因此，对于每个文本可以有多个标签的多类分类，我们可能更倾向于 Noul；而对于只有一个最终答案的分类问题，Choice 提供了更多便利。
Here, each question (independently from each other) returns a probability, and these probabilities don’t have to sum to 1. So, for a multi-class classification where each text can have multiple labels, we may prefer Noul, and for classification problems with only one final answer, Choice adds more convenience.
现在，使用 Choice 或 Noul 在测试集的 25,000 条电影评论上运行 Jev，得到以下结果：
Now, running Jev with either Choice or Noul on the 25,000 movie reviews on the test set gave the following results:
* 15,456,663 个输入 token，总成本 $0.6492
* 15,456,663 input tokens, $0.6492 total cost
* 15,106,663 个输入 token，总成本 $0.6345
* 15,106,663 input tokens, $0.6345 total cost
注意，Choice 和 Noul 在性能和运行时间上的微小差异可能是由于随机波动，因为与其他 LLM 类似，运行并非完全确定性的。
Note that the small difference in performance and runtime between Choice and Noul could be due to random fluctuation since, similar to other LLMs, the runs are not fully deterministic.
例如，我在同一测试集上第二次运行 Choice API，得到了略有不同的结果，如下所示。
For instance, I ran the Choice API a second time over the same test set and got slightly different results, as shown below.
（大规模服务 LLM 及其他模型时出现的不确定性，很可能源于依赖批次的 GPU kernel 执行改变了浮点运算的顺序，正如 Horace He 去年在一篇精彩的博客文章中所阐述的那样。）
(The non-determinism when serving LLMs and other models at scale is likely due to batch-dependent GPU kernel execution changing the order of floating-point operations, as laid out in a nice blog post by Horace He last year.)
无论如何，总体结果看起来相当不错（需注意：我们并不知道 IMDb 测试集是否属于训练集的一部分）。
Anyways, the results look quite good overall (caveat: we don't know if the IMDb test set was part of the training set).
* 在训练集上微调耗时 23 分钟；
* 23 min to fine-tune on the training set;
作为对比，最佳的 ModernBERT 模型具有相近的准确率，如下所示（我预计通过额外的超参数调优还能再提升 1-2% 的准确率）。请注意，该结果是在 DGX Spark 上运行的，而通过量化和更快的硬件有可能获得更快的推理性能。
For comparison, the best ModernBERT model has similar accuracy, as shown below (I expect you can still get 1-2% higher accuracy with additional hyperparameter tuning). Note that this was run on a DGX Spark, and it's possible to get faster inference performance with quantization and faster hardware.
但我们的 ModernBERT 模型不会玩俄罗斯方块，例如，若不针对新任务对它进行微调，它除了电影分类之外也做不了别的事。不过我们随后可以拿一个 GPT-5.6 Luna 或 GPT-6 Luna 模型来应对（它稍慢一些，也更贵）。
But our ModernBERT model can't play Tetris, for example, or do anything else besides movie classification without us fine-tuning it for the new task. But we could then throw a GPT-5.6 Luna or GPT-6 Luna model at it (which is a bit slower and also more expensive).
* 对于一次性的决策任务，使用一个廉价的 LLM（如 GPT-6 Luna）；
* Use a cheap LLM (like GPT-6 Luna) for one-off decision-making tasks;
* 如果我们想反复执行该任务，则微调一个自定义分类器。
* Fine-tune a custom classifier if we want to do this task repeatedly.
现在，像 Jev 这样的东西可以取代上述做法：a) 相比 Luna 降低延迟并节省费用；b) 省去我们微调自定义模型的工作。（当然，如果我们有一个数据量极大的任务，并且想在某个非常具体的任务上最大化速度与准确率，那么微调仍然是有意义的。）
Now, something like Jev could take the place of the above to a) reduce latency and save money over Luna and b) save us the work of fine-tuning a custom model. (Although if we have a very high-volume task and we want to maximize speed and accuracy on a very specific task, it, of course, still makes sense to fine-tune.)
当然，我们也可以在（Modern）BERT 或任何 GPT 风格模型之上添加类似 Jev 的 API。添加类似 Jev 的 API 非常直接。事实上，Jev 一经发布，我就构建了一个类似 Jev 的 ModernBERT 模型，以展示构建自己的 Jev 模型是多么简单。
Of course, we could also be adding a Jev-like API on top of a (Modern)BERT or any GPT-style model. Adding a Jev-like API is pretty straightforward. In fact, as soon as Jev was released, I built a Jev-like ModernBERT model to show how simple it is to build your own Jev model.
我决定不发布我的 Jev 克隆，因为在此期间我改变了主意。也就是说，在 ModernBERT 之上放置类似 Jev 的 API 是微不足道的，在一堆分类任务上微调它也是微不足道的。但是，如果没有大量的训练和测试，要让这个模型在所有不同类型的任务（如俄罗斯方块）上良好运行并非易事。（此外，现在已经有很多快速 Jev 克隆搭上了炒作列车；世界不需要另一个快速克隆，但一个强大的开放权重版本当然会很好。）
I decided not to release my Jev clone because I changed my mind in the meantime. I.e., it’s trivial to put a Jev-like API on top of ModernBERT, and it’s trivial to fine-tune it on a bunch of classification tasks. But it’s not trivial to make this model work well on all different kinds of tasks (like Tetris) without extensive training and testing. (Also, there are already enough quick Jev clones riding on the hype train by now; the world doesn’t need another quick clone, but a strong open-weight version would be nice, of course.)
然而，如果你对这种改造如何工作感兴趣，这里有一个快速概述。例如，我们可以通过向任何 BERT、GPT 或 T5 风格模型添加一个小型分类头来实现类似 Jev 的 Choice API，如前所示。但是，不是让这个头中的输出节点数与类别数匹配，而是只有一个输出节点，如下面修改后的 GPT 模型所示。
However, if you are interested in how that retrofitting would work, here is a quick overview. For example, we can implement a Jev-like Choice API by adding a small classification head to any BERT-, GPT-, or T5-style model as illustrated earlier. But instead of having the output nodes in this head match the number of classes, we have it with only 1 output node, as illustrated for the modified GPT model below.
图 31：GPT 模型，我们将输出层（映射到整个词汇表）替换为只有一个节点的输出头。
Figure 31: GPT model where we replace the output layer (which maps to the whole vocabulary) with an output head with only one node.
之前，我们讨论了针对特定任务（如电影评论分类）微调这些模型，使用预定义的类别标签数（这里为“正面”和“负面”）。通过这种“1 个输出节点”的设置，我们实际上可以将其扩展到灵活且任意的类别数。
Earlier, we discussed fine-tuning these models for a specific task like movie review classification with a pre-defined number of class labels (here, “positive” and “negative”). With this “1 output node” setup, we can actually extend this to a flexible and arbitrary number of classes.
为了将其扩展到任意数量的类别（让我们考虑将客户工单归类为“billing”“technical”“account”三个类别的 3 类情形），我们可以按下图所示的方式实现。
To extend this to an arbitrary number of classes (let's consider the 3-class case of categorizing a customer ticket into the three categories "billing", "technical", "account"), we can do so as shown in the figure below.
图 32：使用类似 Jev 的选择 API 的 BERT 或 GPT 风格模型。
Figure 32: Using a BERT- or GPT-style model with a Jev-like choice API.
如上图所示，对于每个候选选项，我们将输入文本、任务指令以及候选描述一并输入模型。分类（评分）头将输出表示映射为一个标量分数。随后，我们对各候选分数应用 softmax 以获得概率分布，并返回概率最高的选项。
As shown in the figure above, for each candidate option, we feed the model the input text, the task instructions, and the candidate's description. The classification (scoring) head maps the output representation to a single scalar score. We then apply softmax across the candidate scores to obtain a probability distribution and return the highest-probability option.
这里的要点在于，所提出的头对每个类别都有一个输出。每个类别描述都会产生各自的表示，由同一个头通过输出层（本质上是一个逻辑回归模型）对其进行评分：
The main point here is that the proposed head has one output per class. Each class description produces its own representation, which the same head scores via the output layer (which is essentially a logistic regression model):
* \(s_i\) 是类别标签 \(i\) 的标量分数；
* \(s_i\) is the scalar score for class label \(i\);
* _h i_ 是 Transformer 在分类头之前的输出。
* _h i_ is the output of the transformer before the classification head.
对于 _N_ 个候选，这会产生 _N_ 个分数。然后 Softmax 返回 _N_ 个概率。当 _N_ 变化时，学习到的参数保持不变。
For _N_ candidates, this produces _N_ scores. Softmax then returns _N_ probabilities. The learned parameters and stay the same when _N_ changes.
* BERT 风格的编码器模型使用 [CLS] 标记的最终隐藏状态（可选地通过池化层）；
* BERT-style encoder models use the final hidden state of the [CLS] token (optionally passed through a pooling layer);
* GPT 风格的自回归模型通常使用最后一个非填充标记的隐藏状态，由于前面讨论的自回归性质，我们需要这样做（最后一个非填充标记可以关注前面的文本、指令和候选描述）。
* GPT-style autoregressive models typically use the final non-padding token’s hidden state, which we need because of the autoregressive nature as discussed earlier (the last non-padding token can attend to the preceding text, instructions, and candidate description).
然后，我们使用针对正确选项的交叉熵损失来联合微调模型和评分头。由于所有候选共享相同的评分头，我们可以在不改变架构的情况下更改选项的数量和描述。但同样，这在陌生任务上的效果取决于训练数据。
Then, we fine-tune the model and the scoring head jointly using cross-entropy loss against the correct option. Since all candidates share the same scoring head, we can change the number and descriptions of the options without changing the architecture. But again, how well this works on unfamiliar tasks depends on the training data.
顺便问一下，为什么选择 BERT 或 GPT 风格的 Transformer 模型，而不是前面提到的更简单的 RNN 和 CNN 架构？从技术角度看，上述方法对两者都适用。但 Transformer 模型的 Scaling（规模扩张）效果非常好，这意味着它们可以在更大的数据集上进行预训练，并比其他模型获益更多。此外，它们能很好地利用上下文中提供的信息（得益于注意力机制）。因此，如果我们希望模型在不针对每个任务进行显式微调的情况下，也能很好地泛化到不同的目标任务，那么在高质量数据集上预训练一个 Transformer 模型，可能比基于 RNN 或 CNN 的模型更接近这一目标。
By the way, why BERT- or GPT-style transformer-based models over simpler RNN and CNN architectures mentioned earlier? From a technical perspective, the approach outlined above works with either. But transformer-based models scale really well, meaning they can be pre-trained on larger datasets and benefit more than other models. Also, they can make good use of the information provided in the context (thanks to attention). So, if we want our model to generalize well to different target tasks without explicit fine-tuning on each one, pre-training a transformer-based model on a high-quality dataset likely gets us closer than RNN- or CNN-based models.
那么，Jev 为何能在如此多不同的任务上表现优异（从电影评论到像玩俄罗斯方块这样随意的任务）？遗憾的是，其架构、算法和训练数据的细节并未公开。此外，请记住，背后有一整个团队和一家价值数百万美元的公司专门为此模型付出了巨大努力；我们不能指望通过在一些开放数据集上训练一个类似 ModernBERT 的模型一周就能达到那样的性能水平。
So, how come Jev does so well on so many different tasks (from movie reviews to something arbitrary like playing Tetris)? Unfortunately, the architecture, algorithm, and training data details are not public. Also, keep in mind there’s a whole team and multi-million-dollar company behind it that specialized in and worked really hard on this model; we can’t expect to match that level of performance by training a ModernBERT-like model for a week on some open datasets.
话虽如此，如果我必须做出有根据的猜测，在架构方面，我猜他们使用的是类似 ModernBERT 的小型模型，因此延迟较低。
That being said, if I had to make an educated guess, architecture-wise, I’d guess that they are using something small similar to ModernBERT, hence, the low latency.
关于训练数据，如前所述，TypeSafe AI 首席执行官说了以下内容：
For the training data, as mentioned before, the TypeSafe AI CEO said the following:
我们 100% 的数据都是合成的（但显然不是那种由 LLM 直接吐出的垃圾）
100% of our data is synthetic (but not the type of crap that is just spit out from an LLM obviously)
所以，是的，我认为大部分努力都花在了整理这个数据集上。这也是我多年来一直向学生和合作者宣扬的事情。作为一个简短轶事，大约 8 年前，我与社会科学系的另一位教授合作，帮助设计了一个文本分类问题的实验设置。她的学生花了数天时间对词袋基线和 BERT 模型进行超参数调优，勉强提升了约 2-5% 的准确率。然后我建议我们每人花几天时间手动标注更多数据（我记得原始数据集大约有 300 个样本，我们将其规模翻倍），结果准确率提升了超过 10-20%。是的，为此绘制学习曲线很重要：）。
So, yeah, I think most of the effort went into curating this dataset. It’s something I’ve also been preaching to students and collaborators for many years. As a short anecdote, about 8 years ago, I was collaborating with another professor in the social sciences department and helped to design the experimental setup for a text classification problem. Her student spent many days hyperparameter tuning both a bag-of-words baseline and a BERT model to eke out ~2-5% accuracy. Then I suggested we each sit down for a few days to hand-label more data (I think our original dataset was around 300 samples, and we doubled the size), which resulted in a >10-20% accuracy boost. Yes, it’s important to plot learning curves for that purpose :).
图 33：经验法则表明，通常更多数据比额外的超参数调优更有帮助。
Figure 33: Rule of thumb showing that often more data helps more than additional hyperparameter tuning.
最后是训练算法。同样，细节未知，但 TypeSafe AI 的博客文章指出，他们使用了一种名为“校准决策强化学习”（Reinforcement Learning for Calibrated Decisions, RLCD）的新算法：
Finally, the training algorithm. Again, the details are not known, but TypeSafe AI's blog post states that they were using a new algorithm called Reinforcement Learning for Calibrated Decisions (RLCD):
“我们构建了一个完全专注于自动化的新技术栈：采用新的模型架构、用于最大化效率的并行采样器，以及我们称为‘校准决策强化学习’（RLCD）的训练方法。”
"We built a new stack entirely focused on automation: with a new model architecture, parallel sampler for maximum efficiency, and training method we call Reinforcement Learning for Calibrated Decisions (RLCD)."
该方法未公开。一个已发表的具有类似校准目标的方法是 RLCR（Reinforcement Learning with Calibration Rewards），出自 2025 年的论文《超越二元奖励：训练语言模型对其不确定性进行推理》（Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty）。在提供更多细节之前，下一节将简要概述校准的一般概念。
This method is not public. A published method with a similar calibration objective is RLCR (Reinforcement Learning with Calibration Rewards), from the 2025 paper Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty. Before providing more details, the next section gives a brief overview of calibration in general.
校准并非一个新话题，对于任何需要使用并评估类别归属概率的生产模型，我都建议进行这一步。简而言之，校准会调整模型的概率估计，使其更好地匹配观测到的类别频率。
Calibration is not a new topic, and it is a step I recommend for any production model where you want to use and evaluate the class-membership probabilities. In short, calibration adjusts the model's probability estimates so they better match observed class frequencies.
例如，再次考虑我们的 IMDb 电影分类问题。模型可能将一条评论判为 74% 正面、26% 负面。这些是概率估计，但模型可能过度自信或信心不足。若不评估校准，我们就不能假定这些数值是可靠的。也就是说，在 50% 阈值下，74% 正面和 54% 正面的预测都会得到类别标签“正面”，尽管模型在 74% 的情况下表达了更高的置信度。
For example, consider once more our IMDb movie classification problem. A model might assign a review 74% positive and 26% negative. These are probability estimates, but the model may be overconfident or underconfident. We cannot assume that the numerical values are reliable without evaluating calibration. That is, predictions of 74% positive and 54% positive both yield the class label "positive" at a 50% threshold, although the model expresses more confidence in the 74% case.
图 34：校准工作原理示意图。
Figure 34: Illustration of how calibration works.
校准技术使用一个单独的带标签数据集（例如留出的验证集）来调整这些概率估计。成功校准后，在被赋予约 74% 正面概率的众多评论中，大约 74% 应当确实是正面的。如需了解更多信息，我推荐经典的 scikit-learn 文档。
Calibration techniques use a separate labeled dataset, e.g., a held-out validation set, to adjust these probability estimates. After successful calibration, among many reviews assigned a positive probability of approximately 74%, roughly 74% should actually be positive. For more information, I recommend the good old scikit-learn documentation.
对此有许多方法。例如，我在自己的 AI 检测器项目中用过的一种方法是温度缩放（temperature scaling）。温度缩放会在应用 softmax 之前，将模型的 logits 除以一个温度值（\(T\)）。我们在保持模型权重固定的同时，通过最小化校准数据集上的交叉熵损失来学习 \(T\)（一个正数）。大于 1 的温度会降低对最高概率类别的置信度，而介于 0 和 1 之间的温度则会提高它。随后我们将同一温度用于新的预测。由于这种缩放保持了 logits 的排序，当我们选择最高概率类别时，预测类别保持不变。
There are many approaches for this. For example, one that I used in my AI detector project is temperature scaling. Temperature scaling divides the model's logits by a temperature (\(T\)) value before applying softmax. We learn \(T\) (a positive number) by minimizing the cross-entropy loss on the calibration dataset while keeping the model weights fixed. A temperature above 1 reduces confidence in the highest-probability class, while a temperature between 0 and 1 increases it. We then use this same temperature for new predictions. Because this scaling preserves the ordering of the logits, the predicted class remains unchanged when we select the highest-probability class.
Jev 的 RLCD 训练方法仍属专有，但我们可以讨论一个相关思路——**基于校准奖励的强化学习**（Reinforcement Learning with Calibration Rewards，RLCR），它由 2025 年的论文《Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty》提出。（注意，这两种方法之间尚无官方确立的联系，但我推测它们可能相关。）
Jev's RLCD training method remains proprietary, but a related idea we can discuss is _Reinforcement Learning with Calibration Rewards_ (RLCR), which was introduced in the 2025 Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty paper. (Note that there is no officially established connection between the two methods, but I am assuming that they could be related.)
带可验证奖励的强化学习（RLVR）通常对正确答案奖励 1、对错误答案奖励 0。（关于全面的解释与实现，我推荐参阅我的《Build A Reasoning Model（From Scratch）》一书：)。）
Reinforcement learning with verifiable rewards (RLVR) typically rewards a correct answer with 1 and an incorrect answer with 0. (For a comprehensive explanation and implementation, I recommend checking out my Build A Reasoning Model (From Scratch) book :)).
简而言之，RLCR 对不准确的置信度估计额外施加惩罚。
In short, RLCR adds an additional penalty for inaccurate confidence estimates.
因此，在传统的 RLVR 方法中，奖励 _R_ 依据答案正确性取 0 或 1（为简化起见，此处忽略可选的格式奖励与长度惩罚）。
So, in the conventional RLVR method, the reward _R_ is either 0 or 1 based on the answer correctness (we are ignoring an optional formatting reward and length penalty here, for simplicity).
在 RLCR 中，模型生成推理过程与答案，随后进行不确定性分析并给出数值置信度（_q_）。这种修改后的奖励为
In RLCR, the model generates reasoning and an answer, which is followed by an uncertainty analysis and a numerical confidence (_q_). This modified reward is
\(R = c - \left( q - c \right)^{2} ,\)
$R = c - \left( q - c \right)^{2} ,$
其中 _c_ 在答案正确时为 1，否则为 0。例如，一个置信度为 90%（0.9）的错误答案获得的奖励为 -0.81，因为
where _c_ is 1 for a correct answer and 0 otherwise. For example, an incorrect answer with 90% (0.9) confidence receives a reward of -0.81, since
\(0 - \left( 0.9 - 0 \right)^{2} = - 0.81 .\)
$0 - \left( 0.9 - 0 \right)^{2} = - 0.81 .$
在置信度为 20% 时，奖励为 -0.04。而置信度为 90% 的正确答案可获得 0.99。（熟悉校准模型评估的读者可能会注意到，平方误差项即为针对所声明的答案正确概率的 Brier 惩罚。）
At 20% confidence, the reward is -0.04. And a correct answer at 90% confidence earns 0.99. (Readers familiar with evaluating calibrated models may notice that the squared-error term is the Brier penalty for the stated probability that the answer is correct.)
注意，同一个（此处为 Qwen2.5-7B）模型会在其响应中生成不确定性分析和 q。
Note that the same (here, Qwen2.5-7B) model generates the uncertainty analysis and q as part of its response.
不确定性分析是对其答案可能出错之处的书面评估。在回答之后，模型会检查缺失的证据、模糊的措辞、可疑的假设或可能的推理错误。论文中的提示要求它识别具体的不确定性，而不是提出修正。
The uncertainty analysis is a written assessment of where its answer could be wrong. After answering, the model examines missing evidence, ambiguous wording, questionable assumptions, or possible reasoning errors. The paper's prompt asks it to identify specific uncertainties rather than propose corrections.
此外，系统提示明确要求 _q_，一个介于 0 和 1 之间的数字，放在标签内。它表示模型对其答案正确的估计概率。系统在同一响应中按顺序生成所有这些部分。
Also, the system prompt explicitly asks for _q_, a number between 0 and 1 inside tags. It represents the model's estimated probability that its answer is correct. The system generates all these parts sequentially in the same response.
为了说明，模型答案可能如下所示：
For illustration purposes, the model answer could be as follows:
<think>...对问题的推理...</think>
<think>...reasoning about the question...</think>
<answer>positive</answer>
<answer>positive</answer>
该段落提到了一条正面影评，但其与第二段的关联并不明确。
The passage mentions a positive movie review, but the connection to the second paragraph is unclear.
<confidence>0.6</confidence>
<confidence>0.6</confidence>
下图是论文中带有注释的图，展示了该方法在温度缩放（temperature scaling）下相较常规 RLVR 如何改善期望校准误差。
Below is an annotated figure from the paper that illustrates how it improves the expected calibration error over regular RLVR with temperature scaling.
在 HotpotQA 上，与 RLVR 相比，RLCR 将期望校准误差（ECE）从 0.37 降至 0.03，准确率相近（62.1% 对 63.0%）。在另外六个数据集上，平均 ECE 从 0.46 降至 0.21，同时准确率从 53.9% 升至 56.2%。这些是论文表 1(a) 中的结果。
On HotpotQA, RLCR reduces expected calibration error (ECE) from 0.37 to 0.03 compared with RLVR, with similar accuracy (62.1% versus 63.0%). Across six other datasets, average ECE falls from 0.46 to 0.21, while accuracy rises from 53.9% to 56.2%. These are the results in the paper's Table 1(a).
图 36：来自《Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty》论文的 RLCR 结果。
Figure 36: RLCR results from the Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty paper.
从概念上讲，“RLVR + Classifier”中的“Classifier”是一个监督式二分类器，用于预测生成的答案是否正确。
Conceptually, the "Classifier" in "RLVR + Classifier" is a supervised binary classifier that predicts whether the generated answer is correct.
对于 Jev，我可以直接将校准奖励应用于类型化决策，而无需生成推理文本。例如，我们可以为 GPT 风格模型配备第 2.2 节所示的分类头，并调整强化学习训练目标，以奖励准确的决策和校准良好的概率。我们将联合训练骨干网络和分类头。这将是一种受 RLCR 启发的改编，而 Jev 的 RLCD 是否以这种方式工作仍不清楚。
For Jev, I could apply calibration rewards directly to typed decisions, without generating reasoning text. For example, we could equip a GPT-style model with the classification head illustrated in Section 2.2 and adapt the RL training objective to reward accurate decisions and calibrated probabilities. We would train the backbone and classification head together. This would be an adaptation inspired by RLCR, and it remains unclear whether Jev's RLCD works this way.
或者，由于直接策略优化（DPO）是基于人类反馈的强化学习（RLHF）的交叉熵（CE）替代方案，我们也可以直接在分类器的概率上最小化 CE + Brier 损失。我在 ModernBERT 模型上做了这一尝试，相比常规温度缩放有适度改进。
Alternatively, since Direct Policy Optimization (DPO) is a cross-entropy (CE) alternative to Reinforcement Learning with Human Feedback (RLHF), we could also directly minimize CE + Brier loss on the classifier's probabilities. I did this with the ModernBERT model, and there was a modest improvement over regular temperature scaling.
图 37：通过受 RLCR 启发的 Brier 损失优化对 ModernBERT 的改进。↑ 符号表示越高越好。↓ 符号表示越低越好。
Figure 37: ModernBERT improvements via RLCR-inspired Brier-loss optimization. The ↑ symbol means higher is better. The ↓ symbol means lower is better.
（为简单起见，上图仅展示九个数据集上的平均结果。总体而言，可以看出校准在平均意义上是有益的。然而，它在九个数据集中的一个上略微恶化了 ECE。）
(For simplicity, the figure above only shows the average results over nine datasets. Overall, as can be seen, calibration was beneficial on average. However, it slightly worsened the ECE on one out of the nine datasets.)
交叉熵本身已经鼓励准确的概率估计。在理想条件下，这足以获得校准良好的预测，而无需额外的惩罚项。
Cross-entropy already encourages accurate probability estimates. Under ideal conditions, that is enough to obtain calibrated predictions without an additional penalty.
也就是说，用一个类似于电影评论分类的简单抛硬币例子，两个类别是“正面”和“反面”。如果这是一枚有偏的硬币，有 80% 的时间（在期望意义上）落在正面，那么真实概率就是 80% 正面和 20% 反面。在一个具有代表性的训练集上，当我们优化交叉熵时，模型作为最小化交叉熵（负对数似然）的副作用，就已经会学到这些概率。
I.e., using a simple coin flip example analogous to the movie review classification, the two classes are "heads" and "tails." If it's a biased coin that lands heads 80% of the time (in expectation), the true probabilities are 80% heads and 20% tails. On a representative training set, when we optimize cross-entropy, the model will already learn these as a side-effect of minimizing cross-entropy (negative-log-likelihood).
因此，简而言之，如果模型学到了真实的类别概率，其预测就是校准的，额外的惩罚项是不必要的。（而且 Brier 损失与交叉熵具有相同的理论最优值，如 Gneiting 和 Raftery 于 2007 年在《Strictly Proper Scoring Rules, Prediction, and Estimation》中所述。）
So, in short, if the model learns the true class probabilities, its predictions are calibrated and an additional penalty is unnecessary. (And Brier loss has the same theoretical optimum as cross-entropy, as described in Strictly Proper Scoring Rules, Prediction, and Estimation by Gneiting and Raftery, 2007.)
然而在实践中，我们从来不会训练到（期望意义上的）真正最优值，而是在有限数据集上训练。
In practice, however, we never train to the real optimum (in expectation) but train on a finite dataset.
因此，一旦神经网络能正确分类大多数训练样本，它就可以通过变得更加自信来进一步降低训练集上的交叉熵。而这些置信度估计可能无法很好地泛化到测试集和新数据上。
So, once a neural network correctly classifies most training examples, it can further reduce cross-entropy on the training set by becoming more confident. These confidence estimates may then generalize poorly to the test set and new data.
例如，在 *On Calibration of Modern Neural Networks* 一文中，Guo 等人（2017）观察到“神经网络可以对 NLL 过拟合，而不对 0/1 损失过拟合”。
For example, in *On Calibration of Modern Neural Networks*, Guo et al. (2017) observed that "neural networks can overfit to NLL without overfitting to the 0/1 loss."
这意味着，测试准确率仍可能提升，而测试交叉熵却在变差。因此，即便分类准确率持续提高，过拟合也可能体现在概率估计中。
That means test accuracy can still improve while test cross-entropy worsens. So overfitting can show up in the probability estimates even when classification accuracy continues to improve.
加入 Brier 损失会改变训练过程中预测误差的加权方式。但这是否能改善校准，当然需要在留出数据上检验。（我的实验中这一适度改善只是针对该设置的经验观察，我们不应假定加入 Brier 损失总会有所帮助。）
Adding Brier loss changes how prediction errors are weighted during training. But whether this improves calibration needs to be checked on held-out data, of course. (The modest improvement in my experiment is an empirical observation for this setup, and we should not assume that adding Brier will always help.)
在我的实验中（见前图），在交叉熵之外加入 Brier 损失只带来了非常小的额外校准收益。
In my experiments (see previous figure), adding the Brier loss to cross-entropy provided only a very small additional calibration benefit.
但在强化学习语境中，动机更为直接，例如 RLCR 中所描述的情形——我们衡量答案正确性，且并未在最小化交叉熵。因此，在强化学习语境中，Brier 项在答案正确性奖励之外，额外激励模型报告准确的置信度。
But the motivation is more direct in RL contexts, such as described in RLCR, where we measure answer correctness and don't already minimize cross-entropy. So, in RL contexts, the Brier term adds an incentive to report accurate confidence alongside the reward for answer correctness.
既然我们已经较为详细地介绍了 Jev，那么 Jev 面向谁？根据我的评估，其目标受众相当庞大，交集在于这样两类人：a) 希望节省时间、不必为每个决策任务都微调一个自定义分类器的人；b) 希望比使用 GPT-6 及类似 LLM 这类“大杀器”更省钱的人。
Now that we have covered Jev in reasonable detail, who is Jev for? Based on my assessment, it is a pretty big target audience that intersects between people who a) want to save time by not having to fine-tune a custom classifier for every decision task and b) want to save money over using "the big guns" like GPT-6 and similar LLMs.
此外还有许多有趣的用例值得考虑。下面一节列出了一份简短的想法清单。
There are also lots of interesting use cases to think of. In the section below is a short list of ideas.
用例清单似乎无穷无尽。诚然，有一些炫目的例子，比如让它玩《俄罗斯方块》这类电子游戏（正如我在上文早前的视频中所展示的，用以演示该 API 的低延迟）。然而，对大多数人而言，整理邮件是更实用的选择。例如，人们可以把 Jev 用作额外的垃圾邮件过滤器、优先级排序工具等等。
The list of use cases is seemingly endless. Sure, there are flashy examples like having it play video games such as Tetris (as I showed in my earlier video above to demonstrate the low latency of the API). However, sorting emails is a more practical choice for most. For example, one could use Jev as an additional spam filter, prioritization, and so on.
但除此之外，它还可以充当增强 LLM 智能体框架的工具。在这里，它可以被用于：
But beyond things like that, it could also serve as a tool to augment LLM-agent harnesses. Here it could be used as:
* 作为常规 LLM 智能体框架的预筛选器，扫描我们的上下文以检测提示注入；
* a pre-screener for regular LLM-agent harnesses to scan our contexts for prompt injection;
* 为模型选择推理努力程度；
* select the reasoning effort level for a model;
* 从你注册的技能库中选择一个 skill.md；
* select a skill.md from your registered skill library;
* 将其用作评估或自我改进的评判者；
* Use it as a judge for evaluation or self-refinement;
* 查找相关文件以构建上下文。
* Find relevant files for context building.
这份清单几乎无穷无尽。甚至有人在 arXiv 上发布了一篇综述，分析了最近短短几天内涌现出的 2,170 个与 Jev 相关的项目。
The list is practically endless. Someone even posted a survey on arXiv analyzing 2,170 Jev-related projects that have sprung up recently in just a few days.
自 Jev 发布以来，已有数百乃至上千个快速复制的 Jev 克隆项目涌现。我查看过的大多数项目，都只是用 SFT 微调 ModernBERT 或 Qwen 模型并添加一个类似 Jev 的 API 的简单示例。
Since Jev's launch, hundreds, if not thousands, of quick Jev clones have sprung up. Most of the ones I checked out were just quick examples fine-tuning a ModernBERT or Qwen model with SFT and adding a Jev-like API.
据我所知，它们中没有一个能在如此广泛的任务上达到与 Jev 同等的性能水平。将这些项目与 Jev 相比，就像拿 Alpaca（基于原始 LLaMA 开放权重模型早期指令微调的大语言模型）与 GPT-6 相比。诚然，它们在理念上可能运作相似，但在真实世界任务上的效果会因人而异。
Based on what I can tell, none of them achieves the same level of performance as Jev on such a breadth of tasks. Comparing these projects to Jev is like comparing Alpaca (the early instruction-finetuned LLM based off of the original LLaMA open-weight model) to GPT-6. Sure, it may work similarly in spirit, but your mileage will vary on real-world tasks.
无意冒犯，但它们看起来像是为了蹭热度而做的快速项目，我不想在这里专门推荐某一个。
No offense, but they seem like quick projects to jump on the hype train, and I don't want to plug a specific one here.
这里的一个例外可能是 GLiNER 项目，它已经存在了约 3 年，虽然它与 Jev 在根本上并不相同，但可用于类似的任务。然而，根据一项基准测试，Jev 显然更强。
One exception here might be the GLiNER project, which has been around for ~3 years, and while it's not fundamentally the same, it can be used for similar things. However, based on a benchmark, Jev is definitely stronger.
图 38：通过 https://github.com/AbdelStark/jev-benchmarks/blob/main/results/reports/btzsc-pilot-v1.md 对 GLiNER 与 Jev 进行比较
Figure 38: GLiNER vs Jev comparison via https://github.com/AbdelStark/jev-benchmarks/blob/main/results/reports/btzsc-pilot-v1.md
无论如何，对于 Jev 的优秀开放权重替代品，确实存在真实的需求和愿望。原因不一定是成本（因为 Jev 非常便宜），而是隐私。（另外，如果 Jev 在互联网上已经这么快，想象一下在强大的本地硬件上会有多快。）
Anyways, there is definitely a genuine need and desire for a good open-weight alternative to Jev. The reason is not necessarily cost (because Jev is so cheap) but privacy. (Also, if Jev is already that fast over the internet, imagine how fast it could be on beefy local hardware.)
因此，我当然希望 Jev 能迎来一个 DeepSeek（或 Kimi、GLM 或 MiMo）时刻。我相信人们正在为此努力，但这需要一些时间。TypeSafe AI 与一个专门的专家团队合作了数月。期望在一周内复制这种开发和评估工作是不现实的。（然而，由于这是一个比 1 万亿参数模型更小、更高效的模型，从本质上讲，希望不会花那么长时间。）
So, of course I’d like a DeepSeek (or Kimi, or GLM, or MiMo) moment for Jev. I am sure people are working on this, but it will take some time. TypeSafe AI worked on this for many months with a dedicated team of experts. It’s unrealistic to expect to replicate that development and evaluation work in a week. (However, since this is a smaller and more efficient model than a 1-trillion-parameter model, by nature, it hopefully won’t take that long.)
更新 1（太平洋时间 9 月 29 日上午 10:20）：OpenAI 刚刚在其 DevDay 2026 大会（9 月 29 日）上宣布了 Decision API。它似乎是一个类似 Jev 的模型，直接集成到他们的平台中。
Update 1 (29 September, 10:20 am PT): OpenAI just announced the Decision API at their DevDay 2026 conference (29 September). It appears to be a Jev-like model directly integrated into their platform.
Decisions API 通过将 Luna 的智能聚焦于一组特定的用户定义问题（具有有限的预定义答案）来实现实时决策。开发者使用文本或图像提供上下文，并得到可用于分类内容、路由请求或选择智能体下一步行动的答案。
Decisions API enables real-time decision-making by focusing Luna’s intelligence on a specific set of user-defined questions with finite pre-defined answers. Developers supply context using text or images, and get back answers they can use to classify content, route requests, or choose an agent’s next action.
今天以有限预览版提供，计划在未来几天内广泛发布。
Available in limited preview today with a broad release planned in the coming days.
图 39：OpenAI 在 OpenAI Dev Day 上宣布的类 Jev 决策 API。
Figure 39: OpenAI’s Jev-like Decision API announced at the OpenAI Dev Day.
更新 2：一位读者在下方评论中与我分享了对比语言模型项目。它是外观较好的类 Jev 之一。我试用时，其 IMDb 测试准确率仅为 82.90%（Jev 中超过 96.47%），并且未能通过俄罗斯方块测试。
Update 2: A reader shared the Contrastive Language Models project with me in the comments below. It’s one of the nicer-looking Jev-likes. When I tried it, its IMDb test accuracy is just 82.90% (over 96.47% in Jev), and it fails the Tetris test.
另一位读者来信提到了 Laya，另一个强有力的竞争者。它在 IMDb 上表现更好（92.33%的测试准确率），但在俄罗斯方块测试中失败得更严重。
Another reader wrote in about Laya, another strong contender. It did better at IMDb (92.33% test accuracy) but then failed the Tetris test even worse.
所以，我的观点仍然成立。有许多可靠的替代品。但让 Jev 如此受欢迎的不是独特的想法（如前所述，分类器早已存在），而是它在许多不同任务上表现如此出色。据称它唯一表现不佳的任务（一位读者告诉我）是非英语法律评论。（但薄弱的多语言性能可以通过将其连接到一个廉价的翻译模型来规避，例如 GPT-6 Luna，只需很小的延迟开销。）
So, my point still stands. There are many solid alternatives. But what makes Jev so popular is not a unique idea (as mentioned before, classifiers existed for a long time) but the fact that it performs so well at so many diverse tasks. The only task where it allegedly doesn’t do well (as a reader told me) is legal reviews in non-English languages. (But the weak multilingual performance can be circumvented by connecting it to a cheap translation model, e.g., GPT-6 Luna with a small latency overhead.)
总而言之，乍看之下，Jev 似乎并没有提供任何根本性的新东西。毕竟，人们可以说“它不过是一个分类器”（上面套了一个好用的 API）。然而，它在大量任务上都表现得出奇地好。从这个意义上说，它是专用分类器的即插即用版本。
All in all, at first glance, Jev doesn't seem to offer anything fundamentally new. After all, one might say that "it's just a classifier" (with a nice API on top of it). However, it works surprisingly well across a huge range of tasks. In that sense, it's the plug-and-play version of dedicated classifiers.
当然，专家们仍会微调专用模型，但如今要证明微调一个专用模型是合理的，门槛已经高得多，因为只需随手把 Jev 或类似 Jev 的模型用上去，就能得到足够好的结果。
Sure, experts will still fine-tune specialist models, but the bar to justify fine-tuning a specialized model is now much higher, since it's easy to just throw a Jev or Jev-like at it and get good-enough results.
此外，我并不指望 Jev 及类似 Jev 的模型能解锁新能力或解决此前无法解决的任务。然而，如果我们把这类模型纳入智能体框架（agent harness），以辅助昂贵的 GPT-6 或 Opus 5.5 模型在该框架内进行决策，那么未来使用智能体框架可能会变得更快、更便宜。
Also, I don't expect Jev and Jev-likes to unlock new capabilities or solve previously unsolvable tasks. However, if we make such models part of our agent harnesses to aid the expensive GPT-6 or Opus 5.5 models in decision-making within that harness, using agent harnesses could become much faster and cheaper in the future.
如果你觉得这篇文章有用，欢迎成为 Ahead of AI 的付费订阅者。你的支持能帮助我把更多时间投入到这类文章背后的实验和详细插图制作上。
If you found this article useful, consider becoming a paid subscriber to Ahead of AI. Your support helps me spend more time on the experiments and detailed illustrations behind articles like this.
如果你希望自己动手实现这些概念，也可以通过我的书《Build a Large Language Model（From Scratch）》和《Build a Reasoning Model（From Scratch）》来支持我的工作。（如果你已经有书了，留下一条书评也会很有帮助。）
You can also support my work through my books, Build a Large Language Model (From Scratch) and Build a Reasoning Model (From Scratch), if you'd like to implement these concepts yourself. (And if you already have a copy, leaving a review would also help a lot.)