Build note
The first vertical slice: a terminal chat
In Part 1 the model could write plausible names. It could not hold a conversation. Rather than start on CUDA, I built the entire chat loop on the CPU first, badly but end to end.
The reason is a rehearsal. If you rehearse the whole play with the wrong props, you still learn where the scene changes are before anyone builds the stage. The chat loop is that rehearsal. It tells me exactly what the GPU backend will have to serve, and it does so while every step is still readable C.
The corpus is two speakers
A character model needs text that looks like a conversation. I added scripts/tools/prep-chat.py, which turns a tagged transcript into a training file where each line is labelled A: or B:. The committed example lives under data/chat/.
The tags do two jobs. They teach the model that turns alternate, and they give the generator a stopping condition. When the model emits A: after a B: reply, the turn is over. That is all the dialogue structure the model gets. It is not a chat template and it is not an instruction format; it is a side effect of the data.
What src/chat.c actually does
make chat trains the model on the tagged file, then starts a REPL (read-eval-print loop) in the terminal. Each line you type is one turn. The program loads the weights, appends your turn to the retained context, fills the KV cache from that context, and generates until the reply hits a length limit or emits the next speaker tag.
Two pieces of that sentence need unpacking, because both become load-bearing later.
A KV cache is the stored key and value vectors for every token already processed. Attention needs them again for each new token, and recomputing them on every step is wasted work, so the model keeps them. The cache is what makes a long conversation cheaper than re-reading the whole transcript each turn.
The context policy is the rule for what happens when the conversation outgrows the model’s window. The answer here is to drop the oldest complete turns, because half a turn is worse than none. The program prints what it removed, so the truncation is never silent.
GHEDI_DEBUG dumps the token distribution the model chose from, which is the only way to tell “the model is unsure” apart from “the sampler is odd”.
Why this order matters
The obvious order would be to finish the GPU work first and add chat afterwards, because chat is the fun part. I did the opposite, and Rule 3 in ROADMAP.md says why: make one end-to-end vertical slice work before broadening it.
A vertical slice is a thin path through every layer of the system. It is not a complete feature; it is a complete journey. Here the journey is: load weights, read a turn, build context, generate, print a reply. Every later phase has to keep that journey working. When Phase 4 puts the same REPL on CUDA, the only thing that changes is the backend behind it.
There is a cost. The chat model takes about eight minutes to train on the CPU, and the replies are simple. That is fine. The slice is a contract, not a demo.
What it taught me
The lesson I took from this phase is that the destination defines the road. If chat is what the system is for, then chat is what the tests measure, and every backend has to pass through it. Designing the interface first meant the GPU work in Part 3 had a fixed target instead of an abstract “make the forward pass faster”.
It also produced the warm-cache design that Part 5 depends on. Because the REPL already forwarded only the changed part of the context on the CPU, the GPU version had a shape to copy. The optimisation was already in the plan before the hardware arrived.
How this was made
- Written by: Rupesh, from the
ghedi-crepository (src/chat.c,include/chat.h,scripts/tools/prep-chat.py,ROADMAP.mdPhase 1,runbook.md§6), with AI drafting and editing. - Numbers: the training time and the sample behaviour come from
runbook.md§6 andROADMAP.md. - AI-assisted sections: structure, explanations and copy editing.