Andrej Karpathy blog[](https://karpathy.github.io/2026/02/12/microgpt/#) This is a brief guide to my new art project microgpt, a single file of 200 lines of pure Python with no dependencies that trains and inferences a GPT. This file contains the full algorithmic content of what is needed: dataset of documents, tokenizer, autograd engine, a GPT-2-like neural network architecture, the Adam optimizer, training loop, and inference loop. Everything else is just efficiency. I cannot simplify this any further. This script is the culmination of multiple projects (micrograd, makemore, nanogpt, etc.) and a decade-long obsession to simplify LLMs to their bare essentials, and I think it is beautiful 🥹. It even breaks perfectly across 3 columns: * This GitHub gist has the full source code: microgpt.py
核心贡献 · Key contributions
一个 200 行纯 Python 的 GPT 实现，无依赖，涵盖完整训练和推理流程。 A 200-line pure Python GPT implementation with no dependencies, covering full training and inference pipeline.
包含最小自动求导引擎、分词器、类 GPT-2 架构、Adam 优化器和采样。 Includes a minimal autograd engine, tokenizer, GPT-2-like architecture, Adam optimizer, and sampling.
展示了 LLM 的算法本质：下一个词预测、注意力机制和反向传播。 Demonstrates the algorithmic essence of LLMs: next-token prediction, attention, and backpropagation.
展示了一个小模型（4192 个参数）能从 32000 个名字中学习统计模式。 Shows that a tiny model (4,192 parameters) can learn statistical patterns from 32,000 names.
提供了从 micrograd 到生产级 Transformer 的清晰教育桥梁。 Provides a clear educational bridge from micrograd to production-scale Transformers.
局限 · Limitations
纯 Python 标量自动求导导致极慢，不适用于真实数据集。 Extremely slow due to scalar autograd in pure Python; not practical for real datasets.
使用字符级分词器而非 BPE 等子词分词器，效率受限。 Uses character-level tokenizer instead of subword tokenizers like BPE, limiting efficiency.
无批处理、GPU 支持或混合精度，无法扩展到微小玩具模型之外。 No batch processing, GPU support, or mixed precision; cannot scale beyond tiny toy models.
简化架构（RMSNorm、ReLU、无偏置）可能不反映更大模型的最佳实践。 Simplified architecture (RMSNorm, ReLU, no biases) may not reflect best practices for larger models.
仅在小数据集上训练 1000 步，结果远非最先进生成质量。 Training only 1,000 steps on a small dataset; results are far from state-of-the-art generation quality.
论文章节 · Sections（共 21）
概述Overview
数据集Dataset
数据集与任务设定Let there be an input dataset docs: liststr of documents (e.g. a dataset of names)
分词器Tokenizer
让分词器将字符串翻译为离散符号并返回Let there be a Tokenizer to translate strings to discrete symbols and back
自动微分Autograd
参数Parameters
架构Architecture
1) 多头注意力模块1) Multi-head attention block
2) MLP 块2) MLP block
训练循环Training loop
亚当，受祝福的优化器及其缓冲区Let there be Adam, the blessed optimizer and its buffers
序列重复Repeat in sequence
对单个文档进行分词，并在两侧添加 BOS 特殊标记Take single document, tokenize it, surround it with BOS special token on both sides
将词元序列前向传播通过模型，构建直至损失函数的计算图Forward the token sequence through the model, building up the computation graph all the way to the loss.
Adam 优化器更新Adam optimizer update: update the model parameters based on the corresponding gradients.