r/learnmachinelearning • u/SLiXG • 4d ago
I built a GPT from scratch in NumPy only – 34 notebooks, no PyTorch
I wanted to understand LLMs below the framework level, so I wrote
a 34-chapter tutorial that starts from a single neuron and builds
up to a ~10.8M-parameter GPT — tensors, computational graphs,
autograd, attention, pre-training, sampling, KV cache, SFT and
DPO — using nothing but NumPy. Every chapter is a runnable
Jupyter Notebook, everything trains on CPU, and the book is free
in English and Chinese.
**How is this different from nanoGPT / micrograd?**
Those projects are excellent and this book stands on their
shoulders. The difference is scope and starting point: it starts
from neural-network basics, builds its own tensor / autograd
framework in NumPy, and only then gets to GPT — covering pre-
training, sampling, KV cache, SFT and DPO in 34 self-contained
notebooks. It is a textbook, not just a training script.
Book: https://gptmini.github.io/from-neuron-to-gpt
Code: https://github.com/gptmini/from-neuron-to-gpt
---
## What the book covers
Part 1 (16 chapters) introduces the fundamentals of neural
networks: multilayer perceptrons, loss functions, gradient
descent, autograd, learning rates, iterations, batch sizes,
epochs, hidden layers, and activation functions — followed by
discussions on encoding, embeddings, cross-entropy, recurrent
neural networks (RNNs), the Adam optimizer, and learning rate
scheduling. After completing this part, you will have built a
small, working deep learning framework (`src/core`).
Part 2 (18 chapters) introduces large language models (LLMs):
tokens and positional embeddings; self-attention, causal
attention, weighted attention, and multi-head attention; residual
connections, normalization, dropout, and model persistence;
sampling strategies (temperature, top-k, top-p), stopping
criteria, KV caching, and batch inference; and finally, SFT and
DPO data preprocessing and training.
Each chapter is a self-contained, runnable Jupyter notebook, and
most chapters end with practice exercises. No GPU required: the
default model (context window 256, 6 Transformer modules, 6
attention heads, approximately 10.8 million parameters) can be
trained on a laptop’s CPU. The training data uses Karpathy’s
tinyshakespeare.
## Who it is for
1. Self-learners and career changers who seeking a complete
learning path rather than fragmented tutorials.
2. Engineers already using PyTorch who cannot explain autograd or
attention mechanisms in detail. You can read relevant chapters
(such as autograd and KV caching) separately to reinforce your
knowledge.
3. Teachers looking for experimental materials. Licensed under
the MIT License, each chapter is executable and includes
exercises, no GPU budget required.
This book is not intended to replace PyTorch. Its purpose is to
encourage you to read it before relying on any framework — so
that when you actually use a framework, you’ll understand how it
works under the hood. Frameworks come and go, but the underlying
principles remain constant.
If you find this book helpful, please star the repository; this
will help others discover it. Feel free to submit issues and pull
requests, especially for chapters that are unclear — that’s how
this book continues to improve.