You can build and train a small language model yourself, and everything you need to learn how is free. The only cost is a few hours of rented GPUs at the very end, and you can skip even that.
This map puts three well-known resources in the order that makes each one easier. None of them is made by webridgeAI. Every figure below comes from the page it is linked to, read on 14 October 2026.
Why this is possible now
Andrej Karpathy's nanochat repository states the change in one sentence: a model of GPT-2's ability "cost ~$43,000 to train in 2019" and can now be trained "for only $48", which is about two hours on a rented machine with eight H100 GPUs. The same page works the sum out: about $3 per GPU per hour is about $24 an hour for the machine, so two hours is about $48.
That is the last step of this map, not the first. If you start there you will train a model without knowing what any line of the code does.
The map
| Step | What | Time | Cost |
|---|---|---|---|
| 1 | Watch one lecture and type along | An evening or two | Free |
| 2 | Work through seven chapters of code | A few weeks of evenings | Free (the code), a laptop is enough |
| 3 | Train a GPT-2 grade model and talk to it | An afternoon | About $48 of rented GPUs, or free at toy size |
The times in steps 1 and 2 are our estimate, not a figure from the sources.
Step 1: watch one lecture and type along
Let's build GPT: from scratch, in code, spelled out. is a free lecture by Andrej Karpathy, 1 hour 56 minutes long. In it he builds a small GPT in code, following the paper "Attention Is All You Need" and OpenAI's GPT-2 and GPT-3.
How to use it:
- Open an empty Python file next to the video.
- Pause and type every line he types. Do not copy and paste.
- Run the code each time he does, and check that your output matches his.
Karpathy's own note under the video recommends watching his earlier "makemore" videos first if the ideas feel too fast. They are on the same channel.
What you have at the end: a small model that you wrote, and a working picture of tokens, attention and the training loop.
Step 2: work through seven chapters of code
rasbt/LLMs-from-scratch is the code for Sebastian Raschka's book Build a Large Language Model (From Scratch). The repository is free to read and run, and it had about 106,000 stars on the day we read it. The book itself is sold separately; the code can be followed without it, though the book explains each stage.
The seven chapters, in the repository's own words:
- Understanding Large Language Models
- Working with Text Data
- Coding Attention Mechanisms
- Implementing a GPT Model from Scratch
- Pretraining on Unlabeled Data
- Finetuning for Text Classification
- Finetuning to Follow Instructions
Two things from the repository's own notes are worth knowing before you start:
- You need Python, not a research background. The page says the most important prerequisite is "a strong foundation in Python programming". PyTorch experience helps, and Appendix A is a short introduction to it.
- A laptop is enough. The main chapters are "designed to run on conventional laptops within a reasonable timeframe".
What you have at the end: a GPT-style model you built piece by piece, pretrained on a small dataset, then fine-tuned twice.
Step 3: train a GPT-2 grade model and talk to it
Now open nanochat. It covers every stage in one small codebase: tokenization, pretraining, fine-tuning, evaluation and inference. After steps 1 and 2 you will recognise each part.
The paid route, as the repository describes it:
- Rent a machine with eight H100 GPUs from a cloud provider.
- Run the one script that holds the whole pipeline:
bash runs/speedrun.sh- Wait. The page says the run takes about 1.5 hours.
- Talk to your model:
python -m scripts.chat_cliThe free route: the repository includes runs/runcpu.sh, which shrinks the model so that it trains on a CPU or an Apple Silicon laptop in tens of minutes. The page is plain about the trade: "You will not get strong results in this way." It is still the same pipeline, and it costs nothing.
What you will not get
- Not ChatGPT. The result matches GPT-2, a model from 2019. Karpathy describes talking to it as "a bit like talking to a kindergartener".
- Not a product. It will state wrong things with confidence. The repository suggests asking it who you are, to watch it invent an answer.
- Not a fixed price. The $48 is a sum on one day's GPU rental rates. Rates change, and a mistake that leaves a machine running costs more.
- Not on smaller GPUs without changes. The page says GPUs with less than 80GB of memory need the batch size turned down, and a single GPU takes about eight times longer.
Why it is worth an afternoon
A model you have trained yourself stops being a mystery. You will know what a token is because you made the tokenizer, what attention does because you wrote it, and why a model says false things fluently because you watched yours do it.
Sources
- Andrej Karpathy, nanochat, the repository and its README, read on 14 October 2026.
- Andrej Karpathy, Let's build GPT: from scratch, in code, spelled out., YouTube.
- Sebastian Raschka, LLMs-from-scratch, the repository and its README, read on 14 October 2026.