Build note
Eight threads on an 82-SM GPU: my first kernels
Every program I had written until now ran one instruction stream at a time. When it was slow, I made the loop smarter: fewer passes, better cache behaviour, less allocation. A GPU kernel throws that instinct away. You write the body of the loop once, and tens of thousands of threads run that same body at the same time on different pieces of the data.
The closest thing I know is a spreadsheet formula. You write the formula in one cell, then fill the column and let each copy work on its own row. A kernel is that fill-down, except the copies run simultaneously and each one is told its own position in the sheet.
There is a second project behind this one. ghedi-c has a CUDA backend in
src/device_cuda.cu with sixteen kernels, and I wanted to write that code
instead of only reading it. So c-27-gpu-kernal in the C playground is a
sandbox: small kernels, each with a way to check it is correct, all outside the
main build. This note covers the first sitting, from a kernel that prints its own
thread ID to a tiled matrix multiply and a profiler report on the 3090. The
directory spells it kernal, and I have stopped arguing with it.
A launch is a grid of blocks of threads
The first kernel does nothing but say who it is:
__global__ void hello_from_gpu(void)
{
unsigned int global_id = blockIdx.x * blockDim.x + threadIdx.x;
printf(" hello from the GPU | block %u of %u | thread %u of %u | global %u\n",
blockIdx.x, gridDim.x, threadIdx.x, blockDim.x, global_id);
}
__global__ marks a function that the host launches and the device runs. The
launch itself is the triple-angle syntax:
hello_from_gpu<<<1, 1>>>(); // one block, one thread
hello_from_gpu<<<2, 4>>>(); // two blocks, four threads each: eight in total
The two numbers mean blocks per grid and threads per block. Inside the kernel,
threadIdx.x is the thread’s position within its block, and blockIdx.x is
which block it belongs to. Those two values are how a thread knows which slice
of the data is its job. The product of the grid size and block size is the total
number of threads, and I print that as the global ID.
Three details in this tiny example matter later.
Device printf is buffered inside the GPU. Nothing appears on my terminal until
the kernel finishes and the host synchronizes with cudaDeviceSynchronize(). A
kernel with no visible output is often a kernel I never waited for.
Threads print in no guaranteed order, because they finish in no guaranteed order.
Threads do not run one by one. They run in groups of 32 called warps, and the whole group advances together. A block of four threads therefore occupies a warp slot built for 32, and 28 lanes sit idle. That becomes the punchline of this post.
The data has to travel
With plain cudaMalloc, the GPU does not see my process memory. It sees its own
VRAM, and the only way data gets there is an explicit copy. The round trip is six
steps, and every kernel after this one repeats them:
float *h_a = malloc(bytes); // host memory
cudaMalloc(&d_a, bytes); // device memory
cudaMemcpy(d_a, h_a, bytes, cudaMemcpyHostToDevice);
vector_add<<<blocks, threads>>>(d_a, d_b, d_c, n);
cudaDeviceSynchronize();
cudaMemcpy(h_c, d_c, bytes, cudaMemcpyDeviceToHost);
cudaFree(d_a);
My first version of this printed the first ten elements of the result and called it done. The playground version compares all 1,048,576 elements against a plain CPU loop and fails loudly on the first mismatch. That habit came from watching a kernel produce plausible-looking garbage: a wrong index still writes numbers, and the numbers only look wrong if you check them.
The CUDA_CHECK macro in this file wraps every call and aborts with the file and
line on failure. It is borrowed from the real backend, and it exists because
CUDA reports many errors late. A launch that is legal but wrong can return
cudaSuccess and then fail at the next synchronization, so the error you get
points at the wrong line.
One thread per element, and one guard
The vector add is the canonical second kernel:
__global__ void vector_add(const float *a, const float *b, float *c, int n)
{
int i = blockIdx.x * blockDim.x + threadIdx.x;
if (i < n) {
c[i] = a[i] + b[i];
}
}
With one thread per element, the grid has to cover n elements:
const int threads = 256;
const int blocks = (n + threads - 1) / threads;
That is ceiling division. With a thousand integers and 256 threads per block, it
gives four blocks and 1024 threads, so 24 threads have nothing to do. The if
guard is what makes those 24 harmless. Without it they would write past the end
of the array, which CUDA does not stop and does not report.
Two ways to get this wrong are worth naming. A plain n / threads rounds down
and leaves the tail of every array unprocessed. A missing guard rounds up and
corrupts memory next to the array. The pattern above is the version that is
correct for any n.
Two dimensions for a matrix, three for a volume
Matrix multiply needs a row and a column per thread, so the launch grows a second dimension:
int row = blockIdx.y * blockDim.y + threadIdx.y;
int col = blockIdx.x * blockDim.x + threadIdx.x;
dim3 threadsPerBlock(16, 16); // 256 threads
dim3 blocks(32, 32); // a 512 x 512 matrix
The grid is now a rectangle, and each thread computes one output element by
walking the row of A and the column of B. A 3D volume add extends the same idea
one level further: 8 x 8 x 8 threads per block, a 64 x 64 x 64 array, and a
flat index
int index = z * width * height + y * width + x;
This is where the mental model pays off. A 3D array in C is already one long block of memory with an index computed from coordinates, and the GPU thread hierarchy is the same arithmetic with different names.
My verification for the 2D version was weak, and I want to be honest about it. Two all-ones times all-twos matrices make every element of the result identical, so the check proves the kernel runs and lands in bounds, but it cannot tell a correct column walk from a transposed one. Both sum to the same value when every element is the same number. For the 3D add I printed one interior element at (1, 2, 3) instead of only the corner, which at least exercises the flat-index arithmetic.
The naive multiply drowns in memory
The straightforward matrix multiply assigns one thread per output element, and that thread computes the dot product of a row and a column:
float sum = 0.0f;
for (int k = 0; k < N; ++k) {
sum += A[row * N + k] * B[k * N + col];
}
Count the traffic. There are N² output elements, and each one reads N values from A and N from B, so the whole grid reads 2N³ floats from memory for a problem that only has 2N² inputs. At N = 1024 that is about two billion reads against two million stored values. The arithmetic units spend their time waiting for DRAM.
Tiling fixes that with shared memory, which is a small scratchpad on each multiprocessor that every thread in a block can read:
__shared__ float As[TILE_SIZE][TILE_SIZE];
__shared__ float Bs[TILE_SIZE][TILE_SIZE];
for (int m = 0; m < (N / TILE_SIZE); ++m) {
int k = m * TILE_SIZE;
As[ty][tx] = A[row * N + (k + tx)];
Bs[ty][tx] = B[(k + ty) * N + col];
__syncthreads();
for (int n = 0; n < TILE_SIZE; ++n) {
Cvalue += As[ty][n] * Bs[n][tx];
}
__syncthreads();
}
Each block loads one 16×16 tile of A and one of B, synchronizes, computes, and
moves to the next pair of tiles along the inner dimension. Every value loaded
from global memory is now used by sixteen threads instead of one, so global
traffic drops by a factor of the tile size. With TILE_SIZE 16, that is sixteen
times fewer reads, and the two tiles cost 2 KiB of shared memory per block.
The two __syncthreads() calls are not decoration. The first stops a thread from
reading a tile element whose load has not landed yet. The second stops a fast
warp from overwriting a tile while a slower warp is still computing from it.
Remove either one and the result becomes wrong in a way that depends on timing,
which is the worst kind of bug to chase.
This kernel has a real limitation I would rather write down than hide. The loop
bound is N / TILE_SIZE, which only works when the tile size divides N
exactly. At N = 1000, the last eight columns are silently skipped, and because
the tile loads are unguarded, they would also read past the end of the array. The
if (row < N && col < N) check at the end guards the store and nothing else.
Divisible sizes are a fine restriction for a learning kernel. They are not fine
for a library.
The profiler is where the lesson lands
I wrapped the hello launch in an NVTX range and recorded the run with Nsight Compute and Nsight Systems:
nvtxRangePushA("Kernel Launch");
helloFromGpu<<<2, 4>>>();
cudaDeviceSynchronize();
nvtxRangePop();
NVTX is a labelling API. It adds named ranges to the profiler timeline, so the
report says “Kernel Launch” instead of an anonymous bar, and
nsys profile --trace=cuda,nvtx,osrt picks them up. Nsight Systems shows when
things happened. Nsight Compute explains why they cost what they cost.
The ncu report on that eight-thread kernel is the part of this project that
changed how I think about kernels:
| Measurement | Value |
|---|---|
| Grid and block | 2 blocks × 4 threads |
| Streaming multiprocessors on the card | 82 |
| Waves per SM | 0.00 |
| Achieved occupancy | 2.08% |
| Kernel duration | 36.10 µs |
| DRAM throughput | 0.09% |
A wave is one round of resident blocks, so “waves per SM 0.00” means the grid never filled a single round. Occupancy is the share of an SM’s warp slots that are busy. One warp was resident where an SM can hold 48, and one divided by 48 is the 2.08%. The card has 82 multiprocessors, and this kernel occupied at most two of them, with one warp each, for 36 microseconds.
The kernel is still correct. It printed eight lines exactly as asked. That is the lesson: correctness and utilization are separate questions, and my tests only ever answered the first one. Nsight Compute also volunteers the fix, noting that a four-thread block masks 28 of every 32 lanes and that 128 to 256 threads per block is a reasonable starting range.
I would not read much into the 36 microseconds. Eight printf calls in a kernel
of no work is mostly launch overhead, and a kernel that small cannot be fast. The
useful habit is the question, not the number: what fraction of this machine did
that kernel actually use.
What I would tell myself before starting
- Compare against a CPU result. “It ran” is not “it is correct”, and GPU bugs usually produce plausible output rather than a crash.
- Wrap every CUDA call, and call
cudaGetLastError()right after a launch. Most launch mistakes surface one step later than the mistake. - Make block sizes multiples of 32. A 4-thread block wastes 28 lanes of every warp it occupies.
- Keep the
if (i < n)guard even when the array size looks tidy, because the grid is rounded up by design. - Put a
__syncthreads()on both sides of a shared-memory tile. The bug from missing one is intermittent and looks like a hardware fault. - Free every
cudaMalloc. Twenty-four gigabytes is a lot until a loop runs a thousand times. - Read the profiler before optimizing. My guess about where the time went was wrong, and the report cost less than the guess.
What comes next
The next stage is the grid-stride loop, which flips the mapping around: launch a fixed number of blocks, enough to fill the card once, and let each thread stride through several elements. That removes the one-thread-per-element limit, makes huge arrays practical, and is the shape most production kernels use.
After that comes measurement, occupancy, and reductions, and the map ends with a comparison against cuBLAS. That is the humbling stage every CUDA tutorial warns about, and the reason GEMM libraries exist. My tiled kernel prints its own elapsed time and GFLOPS, but I have not tuned it, and rather than quote a number I cannot stand behind, I am leaving that comparison for the stage where I measure it properly.
The thread I want to pull on is the connection back to ghedi-c. Its backend
already contains the kernels I am learning to write, and the same questions apply
to it: what launch shape, how much shared memory, what fraction of the card. The
ROADMAP ends there for a reason. The point of the sandbox is to make the real
code readable, and to give me somewhere to be wrong without touching the build
that trains the model.
The tensor-core post ended with the same sentence in different clothes: the number on the box describes the best case, and the profiler describes your case. Eight threads on an 82-SM card is that sentence at its most literal.
How this was made
- Written by: Rupesh, from the
c-27-gpu-kernalsources (hello kernel, vector add, 2D and 3D indexing, tiled multiply, NVTX experiment) and the Nsight Compute report recorded next to that experiment, with AI drafting and editing. - Numbers: the launch sizes, occupancy, duration and throughput figures come from
05-rupesh-kernal-nvtx/ncu-report.md, recorded on the RTX 3090. No benchmark number here is quoted from memory. - Deliberately left out: GFLOPS for the tiled kernel, because those runs were not kept with the repository.
- AI-assisted sections: structure, technical explanation and copy editing.