GPT from first principles: project foundation

Reframed: 2026-07-12. This is the planning source of truth for the learning project currently named llm-from-scratch. The repository is the executable learning record.

The decision

This project should be built as a sequence of concept-driven notebook labs, not as a Python package that happens to contain book code.

The immediate job is to build the learning environment:

1. recover the useful work already completed in Chapter 2;
2. turn it into notebooks that make the reasoning visible;
3. continue attention through the same notebook method;
4. extract reusable Python only after the notebook work proves what the interface should be;
5. finish with a small, tested GPT and a reproducible training run.

The repository should show the path from intuition to tensors to implementation. Clean code alone would hide the most valuable evidence: how the model was understood.

Project identity

Recommended name

Rename the repository from llm-from-scratch to:

gpt-from-first-principles

Use GPT from First Principles as the human-facing project title.

llm-from-scratch is accurate but generic, and the project is not attempting to implement every kind of large language model. The actual arc is specific: tokenization, embeddings, attention, transformer blocks, a GPT model, pretraining, evaluation, and generation. gpt-from-first-principles states that scope honestly and emphasizes understanding rather than copying.

This is the least expensive time to rename it. There are only a few commits, no published package, no deployment depending on the repository name, and no established external audience. GitHub redirects the old repository URL after a rename, but local remotes and any hard-coded references should still be updated deliberately.

The repository description should be:

Building a small GPT from first principles through executable notebooks, tested components, and reproducible CPU training.

What it is not

This is not an implementation of the Raschka book presented as original research. The book is the guide and should be credited plainly. The original value comes from the experiments, explanations, tests, deviations, mistakes, and evidence produced while working through it.

It is also not the same artifact as serving a quantized GGUF model on devata. The two projects support one career story, but the served model is not the hand-built GPT unless that becomes technically true later.

How the learning loop works

Every concept moves through the same loop:

question
  -> prediction in Pragalva's words
  -> smallest executable example
  -> inspect values and tensor shapes
  -> change one variable
  -> explain what changed and why
  -> implement the concept
  -> test an invariant
  -> decide whether any code is ready to extract

The notebook is not a polished lecture delivered to Pragalva. It is the workbench where Pragalva earns the explanation.

How the mentor helps

The mentor may:

• recover and reorganize existing scripts into notebook cells;
• write the notebook scaffold, diagrams, prompts, assertions, and visualization helpers;
• create small experiments that expose a concept clearly;
• review Pragalva’s explanations and identify mistaken reasoning;
• diagnose errors after Pragalva has attempted the step;
• help extract stable code and write tests once the concept is understood;
• keep the repository consistent and reproducible.

The mentor should not silently fill every reasoning checkpoint, replace an attempted implementation with finished code, or turn the notebook into generated prose. Blank prediction and explanation cells are intentional work, not unfinished documentation.

When Pragalva asks for a direct explanation or implementation, give it. The guardrail protects the default learning loop; it is not a refusal mechanism.

Notebook contract

Every notebook should contain only the sections it needs, in this general order:

1. Question: one concept and the concrete thing being investigated.
2. Prerequisites: earlier notebooks or mathematics actually required.
3. Prediction: a short Markdown prompt answered before execution.
4. Build it in small steps: short code cells with visible intermediate values.
5. Shape ledger: the meaning and shape of important tensors at that point.
6. Perturbation: change one input, dimension, mask, seed, or parameter and predict the effect.
7. Implementation: assemble the concept without hiding it behind a helper too early.
8. Invariant checks: assertions that would fail for a plausible wrong implementation.
9. Explain it back: Pragalva writes what the operation does and why it is needed.
10. Checkpoint: what is understood, what remains uncertain, and what may graduate into reusable code.

Notebook cells should be short enough that their output can be understood without scrolling through a wall of tensors. Fixed seeds are used where comparison matters. Outputs stay only when they are small and form part of the explanation; accidental output and execution noise are cleared before merge.

The notebooks are executable documents, so a clean run from top to bottom is part of their definition of done.

Foundation repository shape

Build this shape now:

gpt-from-first-principles/
├── README.md
├── pyproject.toml
├── uv.lock
├── notebooks/
│   ├── README.md
│   ├── 01-text-foundations/
│   │   ├── 01-tokenization.ipynb
│   │   ├── 02-training-windows.ipynb
│   │   └── 03-token-and-position-embeddings.ipynb
│   └── 02-attention/
│       ├── 01-attention-by-hand.ipynb
│       ├── 02-trainable-self-attention.ipynb
│       ├── 03-causal-attention.ipynb
│       └── 04-multi-head-attention.ipynb
├── data/
│   ├── README.md
│   └── the-verdict.txt
└── .gitignore

Create only the Chapter 2 notebooks and the first attention notebook initially. The remaining attention filenames define the near horizon, not permission to generate four finished notebooks in advance.

Why notebooks/, not experiments/chapter-N/

The current structure records where the material appeared in a book. The new structure records what Pragalva understands.

Concept names remain meaningful after the book is finished. Chapter numbers can still be credited inside notebook introductions, but they should not be the primary navigation. The numbered concept folders preserve a learning order without binding the project permanently to the book’s table of contents.

Why no src/ yet

Creating src/llm_from_scratch/attention.py today would force an interface before attention has been worked through. That is architecture by anticipation.

Add src/gpt_from_first_principles/ when the first notebook contains logic that:

1. is understood well enough to explain;
2. is reused by a second notebook or entry point;
3. has a stable responsibility;
4. can be protected by a focused test.

At that point the repository grows:

src/gpt_from_first_principles/
tests/
configs/
scripts/
results/

Those directories are the earned second stage, not today’s scaffold.

Recovering the existing work

The existing Chapter 2 scripts are raw material, not debris and not finished documentation.

Notebook 1: tokenization

Source material:

• regular-expression splitting;
• vocabulary creation;
• SimpleTokenizerV1;
• unknown and end-of-text tokens in SimpleTokenizerV2;
• GPT-2 BPE through tiktoken.

The notebook should compare the approaches rather than merely run them. Pragalva should predict punctuation handling, unknown-token behavior, and how BPE handles an invented word before inspecting the output.

Notebook 2: training windows

Source material:

• next-token input and target shifting;
• sliding windows;
• stride and overlap;
• GPTDatasetV1 and DataLoader.

The notebook should make each input-target pair visible first, then batch it. The central explanation is why the target is the input shifted by one position and how stride changes sample reuse.

Notebook 3: token and position embeddings

Source material:

• torch.nn.Embedding;
• token embedding lookup;
• positional embedding lookup;
• shape combination.

The notebook should separate token identity from position before adding the two representations. Shape assertions matter more than printing the full embedding matrices.

After these notebooks run cleanly and Pragalva has completed their explanation checkpoints, remove the superseded Chapter 2 scripts in the same branch. Git preserves the old form; the repository does not need two active sources of truth.

Starting attention correctly

The existing attention.py becomes the seed for notebooks/02-attention/01-attention-by-hand.ipynb.

The first notebook should stop before trainable query, key, and value projections. Its job is narrower:

1. define the six input vectors;
2. choose one query vector;
3. predict which inputs should be most related to it;
4. calculate dot-product attention scores explicitly;
5. normalize the scores;
6. verify that the weights sum to one;
7. calculate the context vector;
8. perturb one input vector and explain the changed weights;
9. write the tensor-shape ledger;
10. explain what the context vector represents.

That is enough for one notebook. Trainable self-attention, causal masking, and multi-head attention each deserve their own notebook because each introduces a new idea and a new class of mistakes.

The empty attention.ipynb should not be preserved. It should be replaced by the deliberate notebook rather than filled opportunistically.

The three homes

If changing a file changes an executable explanation or program behavior, it belongs in the repository. If it records intent, a durable decision, or a reusable concept, it belongs in the vault. If it teaches an external reader after the work is complete, it may become a blog post.

Vault plan

For now the vault needs only:

content/devata/planning/journey.md
content/devata/planning/llm-from-scratch-project-overhaul.md

Do not create an LLM note tree while the concepts are still being encountered for the first time.

At a natural notebook breakpoint, list the concepts actually used, check content/devata/dictionary.json, and propose the complete bottom-up note batch. If accepted, concept notes can begin under content/research/llm/<walkthrough>-notes/. A chapter number alone never creates a vault note.

The notebook contains the learning process. A vault concept note contains the durable explanation after the process has converged. A blog post contains the external story written later in Pragalva’s voice. These are three transformations, not three copies.

Branch and merge rhythm

Use a branch for one coherent notebook batch:

foundation/chapter-2-notebooks
attention/attention-by-hand
attention/trainable-self-attention
attention/causal-attention
attention/multi-head-attention

The current attention-is-all-you-need branch should first become the foundation branch because it is the first unmerged branch and the repository structure must change before attention continues. It can be renamed to foundation/notebook-learning-system, or kept if renaming would interrupt current work. The branch name is less important than its scope.

A notebook branch is ready to merge when:

• every notebook runs from a clean kernel, top to bottom;
• the environment can be recreated from the committed manifest and lockfile;
• prediction and explain-it-back checkpoints have been completed by Pragalva;
• assertions protect the important invariants;
• the notebook README links the correct learning order;
• superseded scripts or accidental files are removed;
• the journey receives one concise state update.

Foundation milestone

The foundation is complete when:

• the repository has been renamed to gpt-from-first-principles;
• the root README explains the project, book attribution, learning method, current status, and how to run notebooks;
• pyproject.toml and one lockfile reproduce the environment;
• the three Chapter 2 scripts have become the three text-foundation notebooks;
• the first attention-by-hand notebook has been created from the current experiment;
• Pragalva has run every notebook and completed its reasoning checkpoints;
• all notebooks execute cleanly from fresh kernels;
• the superseded experiments/chapter-2/ and empty notebook are removed;
• no reusable package structure has been invented prematurely.

This milestone establishes how the whole project will be learned. It does not claim that attention is complete.

Later evolution

After the notebook foundation:

1. finish the attention notebook sequence;
2. add GPT architecture notebooks for normalization, feed-forward networks, transformer blocks, and the full model;
3. extract reused implementations into src/ with tests;
4. build training, evaluation, and generation notebooks around thin reusable modules;
5. record a reproducible CPU baseline and honest limitations;
6. close with one clean end-to-end training path and selected career-facing writing.

Experiment trackers, distributed training, a model registry, containers for laptop learning, GGUF conversion, and deployment of the hand-built GPT stay deferred until a completed local experiment creates a real need.

Next move

Do not continue expanding attention.py yet.

First, create the notebook learning system and convert the existing Chapter 2 work together. The mentor scaffolds the notebook structure and recovers the executable code; Pragalva runs it, makes predictions, fills the explanation checkpoints, and challenges anything that does not make sense. Once the three foundations are clean, convert the current dot-product experiment into the first attention notebook.

That is the foundation to build today: not the final library, but the method that will let Pragalva genuinely build it.