The Residual Stream
Build a transformer from the matrix multiply up — in public, from scratch, every week.
The Setup
It's 11 PM somewhere in Kampala.
A student opens a terminal. No Jupyter tutorial. No course to follow. Just a blank .py file, the Chinchilla paper, and a question that won't leave them alone:
"I understand how transformers work. But can I actually build one — from the matrix multiply up — that trains, that converges, that I can debug when it breaks?"
That's what this repo is. Not a collection of finished projects. A live record of the gap between understanding something and being able to build it — closing, week by week, in public.
The target: frontier ML research intern level. The method: implement everything from scratch, read the primary literature cold, run real experiments, and write up what was found — including what broke.
This is the residual stream. Every week, the weights update.
What's Being Built
the-residual-stream/
│
├── lectures_exercises/ Lecture notes and JAX challenge work
│ ├── lectures/ Course lecture material
│ └── exercises/ Notebook-based practice and challenges
│
├── research_papers/ Paper PDFs and primary reading material
│
├── 🧮 foundations/ Mathematical bedrock. No ML yet.
│ ├── linear_algebra/ SVD, eigendecomposition, projections
│ ├── calculus/ Jacobians, backprop by hand, chain rule
│ └── probability/ KL divergence, MLE, Bayesian reasoning
│
├── ⚙️ implementations/ Papers → running code. JAX only. No shortcuts.
│ ├── 01_bpe_tokenizer/ Byte Pair Encoding from the 1994 paper
│ ├── 02_transformer/ Attention Is All You Need, implemented
│ ├── 03_training_loop/ Forward pass → loss → backprop → Adam
│ ├── 04_lora/ Low-rank adaptation from Hu et al. 2021
│ └── 05_chinchilla_laws/ The scaling calculator that broke GPT-3
│
├── 📄 paper_notes/ Structured notes on every paper read
│
├── 🔬 experiments/ Original questions. Stated hypotheses. Real results.
│ ├── 01_tokenization_tax/ Does Luganda really cost 3× more tokens?
│ └── 02_warmup_ablation/ How bad is training without LR warmup?
│
├── 📓 weekly_builds/ One complete, working thing per week
│
└── 📋 LEARNING_LOG.md Honest weekly updates. What broke. What I don't know yet.The 8-Week Arc
Weeks 1–2 — Foundations
Weeks 3–4 — Read the Canon
Weeks 5–6 — Original Experiments
Weeks 7–8 — Push the Frontier
The Principles
Build before you explain.
Every folder in implementations/ started as a blank file, not a copy of someone else's code. The point is not the output. The point is the hour where nothing works and you have to figure out why.
State the hypothesis before running.
Every folder in experiments/ has a README written before the notebook. What am I expecting? Why? What would change my mind? This is what research actually is.
Commit the confusion, not just the clarity.
LEARNING_LOG.md tracks what I don't understand yet. A portfolio that only shows finished, polished work is a highlight reel. This is the full game.
Read the source, not the summary.
Every paper in paper_notes/ was read in full — abstract, methods, experiments, limitations. Blog posts are for after.
Currently In The Stream
WEEK 1 — ACTIVE
- Building: BPE tokenizer + full training loop in JAX
- Reading: Vaswani et al. 2017
- Blocked on: causal masking shape in multi-head attn
- Resolved: in progress
Full weekly updates in LEARNING_LOG.md
The Stack
The rule: implementations use only JAX and NumPy. No PyTorch. No HuggingFace. No library that hides what's actually happening. The constraint is the point.
The Paper Notes Format
Every paper gets the same structure. No exceptions.
The one-sentence claim
In my own words, not the abstract's.
Why it matters
What changed after this paper?
The key experiment
The single result that makes or breaks the argument.
What I found non-obvious
The thing that confused me and how I resolved it.
What I'd test next
If I had compute, what experiment would I run?
Implementation
Link to code if I built something from this paper.
The last two sections are the ones that matter. Reading comprehension is table stakes. Research taste is the thing being built.
Benchmarks I'm Measuring Myself Against
These are the questions a frontier lab research intern should be able to answer cold. Checking these off as the answer becomes automatic — not recited.
| Question | Status |
|---|---|
| Implement backprop by hand for a 2-layer network | in progress |
| Explain why Adam converges faster than SGD with momentum | in progress |
| Read a JAX training loop and spot a bug in 5 minutes | ⬜ not yet |
| Explain what a gradient checkpoint is and when to use it | ⬜ not yet |
| Derive attention from query-key-value formulation | ⬜ not yet |
| Explain the Chinchilla optimal ratio without notes | ⬜ not yet |
| Implement LoRA and explain the rank decomposition | ⬜ not yet |
| Read a new paper and identify the weakest assumption | ⬜ not yet |
Why In Public
There's a version of this where the work happens privately, and a polished portfolio appears when it's "ready."
That version is comfortable and useless.
Building in public creates a forcing function. The README has to be clear because someone else might read it. The experiment has to have a real hypothesis because it's written down before running. The learning log has to be honest because there's no point otherwise.
It also means that if someone working on similar problems finds this repo — at Anthropic, DeepMind, Masakhane, or a research group at Makerere — there's something real to look at, engage with, and push back on.
That's the goal. Not the performance of learning. The actual thing.
Get In Touch
If something here is wrong, I want to know. If you're working on similar problems, I want to talk. If you're at a lab and something here is interesting, reach out.