Build note
The GPU is 4.5x slower, and the profile says why
On the real chat model the GPU took 1231 milliseconds to answer a six-turn script where the CPU took 250. That is 4.9 times slower end to end, and 4.5 times slower per token on the forward benchmark. The useful part of this phase was finding out exactly where the time went.
The comparison I keep coming back to is an X-ray before surgery. You can have a strong opinion about where the pain is. The image either confirms it or sends you somewhere else, and either way you operate with better information.
The benchmark
make chat-gpu runs the Phase 2 REPL on the CUDA backend. The KV cache stays warm between turns, and the code forwards only the newly appended context, using chat_common_prefix to find the suffix whose bytes still match the position-indexed cache. GHEDI_CHAT_NOWARM=1 restores the old full replay so the two paths can be compared, and GHEDI_CHAT_TIMING=1 prints a timing per turn.
make chat-bench runs a fixed six-turn script on both backends. On weights-chat.txt, which is 221,824 parameters with a 256-token window:
| Measurement | CPU | CUDA |
|---|---|---|
| Prompt token | 0.245 ms | 1.228 ms |
| Generated token | 0.296 ms | 1.411 ms |
| Six-turn total | 249.7 ms | 1231.3 ms |
A separate forward-only benchmark gives 0.268 ms/token on the CPU and 1.202 on CUDA. The gap is consistent, so it is not a chat-loop artefact.
What nsys found
nsys is NVIDIA’s system profiler. It records every CUDA call and every kernel with timestamps, then lets you total them afterwards. It runs unprivileged, which matters because the deeper ncu profiler did not.
The report showed the top stall clearly. Each token made 130 cudaMemcpy calls, and those copies consumed 83.8% of all CUDA API time, against 0.429 ms/token of actual kernel execution. The weights were already resident on the device and stayed there. What was crossing the bus on every single token was the activations.
That reframed the problem. The kernels were not slow. The model was spending most of its time handing small arrays back and forth across PCIe, which is exactly what a GPU is worst at.
The change I shipped, which did nothing
My first instinct was to cache the GEMM choice. The code was calling getenv("GHEDI_GEMM") on every matrix multiply, and that is a string lookup in a hot loop. I cached it.
It made no measurable difference. I recorded that, and I did not pretend otherwise. The environment lookup was noise next to 130 memory copies.
The real fix was larger than a one-line change: keep every per-position intermediate on the device, copy back only the logits that host-side sampling needs, fuse the Q/K/V projections into one launch, and batch the prompt positions into matrix-matrix products. Those are structural changes, so they went to Phase 10 with this profile as the evidence.
That is the discipline the roadmap asks for. Phase 5’s exit criterion was a profile and a ranked handoff, not an optimization. Writing down “I changed almost nothing, and here is why” is a result.
The deep profiler was blocked on this host by ERR_NVGPUCTRPERM, a permissions issue with the GPU performance counters. I could have worked around it with sudo, but sudo clears the environment variables the run needs. The nsys data was enough to rank the work, so I left ncu for a later phase where it was actually needed.
Checkpoints, the quieter half
The same phase added resumable checkpoints, and they matter for every later training run.
--save-checkpoint FILE, --checkpoint-every N and --resume FILE write and read a GHEDICK1 binary. The checkpoint holds the weights, the Adam moments, the completed step, the total-step target, the random-number-generator state, and the architecture and tokenizer metadata. A resume refuses a mismatched file rather than silently loading the wrong model.
The important detail is that --steps means the total run length, not the length of this segment. Resume at step 1,000 with --steps 100000 and the training continues to 100,000 with the cosine learning-rate schedule it always had. --stop-at N ends a segment early while keeping that schedule intact, so a short segment is numerically a prefix of the full run.
make checkpoint-test proves it: train N steps, save, resume to M, and compare against one uninterrupted fixed-seed run of M steps. They agree bit for bit.
What it taught me
Profile before you optimize, and be willing to have your instinct corrected. I was sure the getenv call mattered. It did not. The profile said the bus was the problem, and the fix I eventually wrote was the one the profile pointed at, not the one I guessed at.
How this was made
- Written by: Rupesh, from the
ghedi-crepository (src/chat.c,src/bench.c,src/checkpoint.c,docs/10-gpu-kernels.md,docs/17-profiling.md,ROADMAP.mdPhase 4), with AI drafting and editing. - Numbers: the benchmark table and the
nsyspercentages come fromROADMAP.mdPhase 4 anddocs/10-gpu-kernels.md. - AI-assisted sections: structure, explanations and copy editing.