microgpt

Author: Andrej Karpathy | Published: 2026-02-12 | Generated: 2026-03-01 | Domain: karpathy.github.io Tags: ‘#llm’ ‘#gpt’ ‘#transformers’ ‘#python’ ‘#autograd’ ‘#machine-learning’

---

TLDR

microgpt is a single, dependency-free Python file of roughly 200 lines that implements the complete algorithmic loop of a GPT: character tokenization, scalar autograd, a GPT-2-like Transformer, Adam optimization, training, and autoregressive sampling. Trained for 1,000 steps on 32,033 names, its 4,192-parameter model reduces loss from roughly 3.3 (random prediction over 27 tokens) to about 2.37 and generates plausible new names. The project argues that production LLMs preserve this same core structure; their added complexity is primarily the engineering required to scale data, compute, architecture, training, and serving.

Key Takeaways

• Full GPT in ~200 lines: The script includes the dataset loader, character-level tokenizer, Value-based reverse-mode autodiff, Transformer architecture, Adam optimizer, training loop, and inference loop with no third-party dependencies.
• Tiny but functional model: It uses a vocabulary of 27 tokens (a–z plus BOS), 16-dimensional embeddings, 4 attention heads, 1 Transformer layer, a 16-token context window, and 4,192 parameters.
• Training objective: Each name is wrapped as [BOS, ..., BOS]; the model predicts each next token using averaged next-token cross-entropy loss. After 1,000 single-document steps, loss drops from approximately 3.3 to 2.37.
• Attention made explicit: Because microgpt processes one token at a time, it explicitly constructs a live KV cache during both training and inference; cached scalar Value nodes remain connected so gradients flow through prior positions.
• Scale changes implementation, not the core loop: Production LLMs replace scalar Python operations with tensor/GPU execution and add larger datasets, BPE tokenization, deeper/wider networks, RoPE, GQA, MoE, distributed training, quantization, and serving infrastructure—but still predict the next token.

Images & Media

• microgpt source rendered as a three-column triptych — Illustration showing the complete microgpt implementation laid out across three columns.
• RSS icon — Blog RSS feed icon.

Referenced Links

• microgpt.py GitHub Gist — Full source code for the 200-line implementation.
• microgpt web page — Browser-accessible version of the script.
• Google Colab notebook — Hosted notebook for running and experimenting with microgpt.
• micrograd video — Karpathy’s 2.5-hour explanation of autograd and backpropagation.
• tiktoken — Production tokenizer library cited as an example of subword tokenization.
• build_microgpt.py — Progressive implementation history, from bigram counts through the final Adam-based GPT.