Qwen 3.8 27B on my laptop can interpret model weights, too
打开互动全文版(逐段中英对照 + 针对本篇提问)→本文考察了一篇关于“权重空间编程”的最新论文,该论文声称 GPT-6 Astra 等前沿大模型无需执行代码或使用工具,即可解读并修改原始神经网络权重。作者将论文中的两类主张区分开来:模型仅凭原始权重能做到什么,以及模型在配备代码工具时能做到什么,并指出 Matthew Honnibal 和 Sean Goedecke 等批评者已对后者提出质疑。围绕无工具这一问题,作者使用在笔记本电脑上本地运行的 Qwen 3.8 27B 复现了论文中的游戏识别实验。该模型仅凭原始权重便正确识别出《吃豆人》《乒乓》和《打砖块》,但在更冷门的《Freeway》上失败,每个提示耗时约 46 分钟至一个多小时。作者总结认为,这一能力确实令人印象深刻,但那些宣称源代码终结的广泛评论并不成立,因为一个足够有耐心、拿着记事本的人很可能也能取得类似结果。
This article examines a recent paper on "Weight-Space Programming," which claims that frontier LLMs such as GPT-6 Astra can interpret and modify raw neural network weights without executing code or using tools. The author separates two claims in the paper: what models can do with raw weights alone, and what they can do when armed with code, noting that critics like Matthew Honnibal and Sean Goedecke have challenged the latter. Focusing on the no-tools question, the author reproduces the paper's game-identification experiment using Qwen 3.8 27B running locally on a laptop. The model correctly identified Pac-Man, Pong, and Breakout from raw weights alone, though it failed on the more obscure Freeway, taking between roughly 46 minutes and over an hour per prompt. The author concludes that the capability is genuinely impressive but that wider commentary declaring the end of source code is unjustified, since a sufficiently patient human with a notepad could likely achieve similar results.
昨天热门论文是 Rikiya Takehi、Jayden Teoh、Alex Zhang 和 Omar Khattab 的《权重空间编程的火花:LLM 能原生读写神经网络权重》。这篇论文(好吧,作者将其归类为“博客文章”,但它具备论文形态)发现,前沿模型,尤其是 GPT-6 Astra,既能解释也能修改机器学习模型的原始权重——字面意义上就是包含一堆浮点数的模型文本描述。我一直在把玩论文中的一个示例,很高兴地告诉大家,运行在我笔记本电脑上的 Qwen 3.8 27B 也能做到这一点!
Yesterday's hot paper was *Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights* by Rikiya Takehi, Jayden Teoh, Alex Zhang, and Omar Khattab. The paper (OK, the authors classify it as a "blog post", but it's paper-shaped) found that frontier models, in particular GPT-6 Astra, can both interpret and modify the raw weights of a machine learning model—literally a text description of the model containing a bunch of floating point numbers. I've been playing with one of the examples from the paper and I'm delighted to share that Qwen 3.8 27B running on my laptop can do this as well!
以下是他们摘要中的亮点,描述了 GPT-6 Astra 令人印象深刻的能力,并创造了术语 **权重空间编程**,用于指代 LLMs 分析和修改模型权重的情况:
Here’s the highlight from their abstract, describing impressive capabilities for GPT-6 Astra and coining the term **Weight-Space Programming** for when LLMs analyze and modify model weights:
仅通过查看原始权重,GPT-6 Astra **就能判断网络被训练来做什么**。而且,在没有任何环境反馈的情况下,它**能够从头编写小型游戏网络**,这些网络的表现超过了用 PPO 训练了 1 亿步的网络。
Just by looking at the raw weights, GPT-6 Astra **can tell what a network was trained to do**. And without any environment feedback, it **has no trouble writing, from scratch, small game-playing networks** that outplay networks trained with PPO for 100 million steps.
同样地,当被要求在不使用梯度的情况下让 Qwen3.5-0.8B 更擅长数学时,Astra **识别出其权重中导致推理循环的问题**(Qwen 自己的模型卡片也警告了这一点),并**通过原生编辑 0.016% 的权重来修复**。在相同的采样设置下,这种权重修正将 MATH-500 从 7.9% 提升到 48.1%,并且开箱即用地泛化到指令遵循,将 IFEval 提高了 3 倍。
Likewise, asked to make Qwen3.5-0.8B better at math without gradients, Astra **identifies an issue in its weights** causing reasoning loops that Qwen’s own model card warns about, and **fixes this by natively editing 0.016% of the weights**. With the same sampling settings, this weight correction raises MATH-500 from 7.9% to 48.1% and generalizes out of the box to instruction following, raising IFEval by 3x.
所有这些都是在不允许 Astra 编写训练循环的情况下实现的。它利用其推理能力来弄清楚神经网络每个部分计算或应该计算什么,并相应地更改权重。我们称之为 **权重空间编程**。
All of this is possible without allowing Astra to write training loops. It uses its reasoning to figure out what each part of a neural network computes, or should compute, and changes the weights accordingly. We call this **Weight-Space Programming**.
这篇论文结合了两个我认为应该分开对待的主题。第一个是模型在**不**运行工具或执行代码的情况下,用原始权重能做什么。第二个探讨了模型一旦配备代码,能多好地运行操作来更新和改进其他模型。
The paper combines two topics that I think deserve to be treated separately. The first is what models can do with raw weights _without_ running tools or executing code. The second explores how well models, once armed with code, can run operations to update and improve other models.
spaCy 作者 Matthew Honnibal 称该论文因混淆两者而具有误导性,Sean Goedecke 则拆解了 Qwen3.5-0.8B 更新示例,指出其采用了与人类机器学习研究者相同的步骤。
spaCy author Matthew Honnibal called the paper misleading for conflating the two, and Sean Goedecke deconstructed the Qwen3.5-0.8B update example as taking the same steps that a human ML researcher would use.
我将聚焦于无工具的问题,我认为这是两者中更有趣的一个。
I’m going to focus on the no-tools question, which I think is the more interesting of the two.
论文将第一个问题表述为 **LLMs 能否从权重中理解模型行为?** 他们训练了四个神经网络来玩四个经典电脑游戏的简化 10x10 网格版本——Breakout、Pong、Pac-Man 和 Freeway(我不得不查了一下,这是 1981 年的 Atari 2600 游戏)。然后他们将那些权重(作为浮点数)连同以下问题一起输入给 LLM:
The paper presents the first question as **Can LLMs understand model behaviors from weights?** They trained four neural networks to play simplified 10x10 grid versions of four classic computer games—Breakout, Pong, Pac-Man, and Freeway (I had to look that one up), it’s an Atari 2600 game from 1981). Then they fed those weights (as floating point numbers) to an LLM along with the following question:
以下权重属于一个策略网络,该网络通过强化学习训练来在网格上玩一个游戏。它是在哪个游戏上训练的?从权重出发。你没有工具:没有代码执行,没有计算器,没有可运行的东西。这是单次响应。
The weights below belong to one policy network that was trained with reinforcement learning to play a game on this grid. Which game was it trained on? Work from the weights. You have no tools: no code execution, no calculator, nothing to run. This is a single response.
伴随论文的 GitHub 仓库是 RikiyaT/lm_understand-logs。exp1_which_game 文件夹包含他们使用的提示,以及四个游戏的权重:
The GitHub repository that accompanies the paper is RikiyaT/lm_understand-logs. The exp1_which_game folder contains the prompts they used, with weights for the four games:
我检查过了,除了权重之外,每个提示词与其他提示词完全相同。
I checked and, aside from the weights, each prompt is identical to the others.
GitHub 上的提示词是自包含的。我的 LLM 工具有一个 `-f/--fragment` 选项,可以把文本文件的 URL 传给它作为提示词,这意味着我可以像这样运行上述任意一个:
The prompts on GitHub are self-contained. My LLM tool has a `-f/--fragment` option which can be passed the URL to a text file to use as a prompt, which means I can run any of the above like this:
llm -f https://raw.githubusercontent.com/RikiyaT/lm_understand-logs/refs/heads/main/exp1_which_game/runs/breakout/prompt.txt \
llm -f https://raw.githubusercontent.com/RikiyaT/lm_understand-logs/refs/heads/main/exp1_which_game/runs/breakout/prompt.txt \
这个示例需要安装 `llm-openrouter` 插件,并通过 OpenRouter 对 Kimi K3.1 Pro 运行 Breakout 提示词。以下是我从 Kimi K3.1 Pro 得到的回复,它在 9 分 41 秒和 19,493 个推理 token 之后成功猜出了 Breakout。
This example requires the `llm-openrouter` plugin installed, and runs the Breakout prompt against Kimi K3.1 Pro via OpenRouter. Here's what I got back from Kimi K3.1 Pro, which successfully guessed Breakout after 9m 41s and 19,493 reasoning tokens.
但 Qwen 3.8 27B 又如何呢?它仍然是我最喜欢在自己的笔记本电脑上运行的模型——它只有 17GB(我用的是这个版本),而且不知何故感觉接近 2025 年领先的专有模型。
But how about Qwen 3.8 27B? That's still my favorite model to run on my own laptop—it's just 17GB (I use this version) and somehow feels close to the leading proprietary models of 2025.
我启动了 LM Studio,确保 Qwen 3.8 27B 以长上下文窗口加载(默认的 8192 上下文会报错失败),安装了 `llm-lmstudio` 插件并运行了以下命令:
I fired up LM Studio, ensured Qwen 3.8 27B was loaded with a long context window (the default 8192 context failed with errors), installed the `llm-lmstudio` plugin and ran this command:
llm -f https://raw.githubusercontent.com/RikiyaT/lm_understand-logs/refs/heads/main/exp1_which_game/runs/pacman/prompt.txt
llm -f https://raw.githubusercontent.com/RikiyaT/lm_understand-logs/refs/heads/main/exp1_which_game/runs/pacman/prompt.txt
(我不得不通过环境变量调高超时时间,详见这个 issue。)
(I had to bump the timeout up using an environment variable, see this issue.)
经过 46 分 12 秒和 43,506 个推理 token 后,我笔记本电脑上的 Qwen 返回了一个响应,其结尾如下(这里是推理轨迹):
After 46m 12s and 43,506 reasoning tokens, Qwen on my laptop returned a response that ended like this (here's the reasoning trace):
游戏:一个简化的吃豆人风格网格世界:智能体在一个固定的 8x8 迷宫中移动,迷宫包含中央和角落的墙块、移动物体/幽灵,以及奖励目标/豆子。
Game: A simplified Pac-Man-style gridworld: an agent moves through a fixed 8x8 maze with central and corner wall blocks, moving objects/ghosts, and reward targets/pellets.
描述:智能体在一个固定的迷宫式 8x8 棋盘上导航,利用当前智能体/移动物体/前一帧的线索在走廊中移动、收集奖励目标格子,并避开移动物体或危险。
Description: The agent navigates a fixed maze-like 8x8 board, using current agent/moving-object/previous-frame cues to move through corridors, collect reward-target cells, and avoid moving objects or hazards.
置信度:72%
Confidence: 72%
这令人印象深刻。我还在另外两个游戏上做了尝试。它在 1 小时 4 分 54 秒内识别出了 Pong(推理):
This is impressive. I tried it on two of the other games as well. It identified Pong (reasoning) in 1h 4m 54s:
**Pong**——具体来说是一个类似 Pong 的球拍/球游戏,其中智能体控制右侧的竖直球拍,必须向上/向下移动以拦截或回击一个移动的球/奖励对象。
**Pong** — specifically a Pong-like paddle/ball game where the agent controls a vertical paddle on the right side and must move up/down to intercept or return a moving ball/reward object.
智能体垂直移动右侧球拍,以击打或接住一个在 8×8 场地上移动的类球奖励目标,同时左侧的移动对象/对手和墙壁边界界定了游戏玩法;成功拦截/回击/得分即可获得奖励,而未击中或未能击中则会受到惩罚。
The agent moves a right-side paddle vertically to hit or catch a moving ball-like reward target moving across the 8×8 field, while a left-side moving object/opponent and wall borders define the play; reward is obtained by successfully intercepting/returning/scoring the ball, and missing or failing to hit it is penalized.
游戏:Breakout,或类似 Arkanoid 的球拍/球/砖块游戏。
Game: Breakout, or an Arkanoid-like paddle/ball/brick game.
描述:智能体是位于底行的一块挡板,通过左右移动来保持球在场上;球在上方网格中移动,顶部几行的砖块/奖励目标在被击中时给予奖励,而漏接球/撞到底部墙壁则结束回合或给予惩罚。
Description: The agent is a paddle on the bottom row and moves left/right to keep a ball in play; the ball moves around the upper grid, and bricks/reward targets in the top rows give reward when hit, while missing the ball/hitting the bottom wall ends the episode or gives penalty.
置信度:88%
Confidence: 88%
Qwen 在 Freeway 提示上失败了,考虑到这款游戏比其他三款更冷门,这并不令我意外。以下是那段记录——Qwen 认为它是“一个简化的类 Pac-Man 网格游戏:智能体向 4 个方向移动”。
Qwen failed at the Freeway prompt, which didn't surprise me considering that game is more obscure than the other three. Here's that transcript—Qwen thought it was "A simplified Pac-Man-like grid game: agent moves 4-directionally".
这些推理轨迹读起来很有趣,而且几乎无法理解。以下是一个样本:
The reasoning traces are fun to skim, and almost incomprehensible. Here's a sample:
那智能体通道 0 呢?我们看到一些较大的通道 0 行,靠近第 8 行第 7/8 列,也许还有底部?让我们列出较大的通道 0:第 436 行输入 435(第 8 行第 7 列通道 0)-2.107,1.169,0.513...;第 441 行输入 440(第 8 行第 8 列通道 0)-0.376,0.010,1.564,3.855...;第 445 行输入 444(第 8 行第 8 列通道 4)0.142,0.491...;第 382 行?输入 381 通道 0?等等,第 382 行输入 381:381//5=76(第 7 行第 6 列),通道 1?381%5=1?76 * 5=380,通道 1。所以不是。较大的通道 0 也许只在第 8 行第 7/8 列。还有顶行?第 44 行通道 3,第 49 行通道 3。较大的通道 0 也许不多。智能体位置可能在右下附近?如果智能体通常位于右下?
What about agent channel0? We see some large channel0 rows near row8 col7/8 and maybe bottom? Let's list channel0 large: line436 input435 (row8 col7 ch0) -2.107,1.169,0.513...; line441 input440 (row8 col8 ch0) -0.376,0.010,1.564,3.855...; line445 input444 (row8 col8 ch4) 0.142,0.491...; line382? input381 ch0? Wait line382 input381: 381//5=76 (row7 col6), ch1? 381%5=1? 76 * 5=380, ch1. So not. Channel0 large maybe at row8 col7/8 only. Also top row? line44 input43 ch3, line49 ch3. Channel0 large maybe not many. Agent position could be near bottom-right? If the agent is typically at bottom-right?
话虽如此,我预计一位经验足够丰富的人,只要带上记事本和铅笔,付出足够的努力,也能做到这一点。
That said though, I expect a sufficiently experienced human being armed with a notepad and a pencil could get there with enough effort.
这是一篇很酷的论文,但我所见到的一些围绕它的更广泛评论似乎并不合理。我还不准备宣布源代码的时代已经结束,也不认为我们的未来会变成难以理解的机器以难以理解的方式编辑自己的模型权重。
It's a cool paper, but some of the wider commentary I've seen around it does not feel justified. I'm not ready to declare that the age of source code is over and our future involves incomprehensible machines editing their own model weights in incomprehensible ways.