CUDA Programming in 21 Days: A Hands-On Course in C++ and Python (Learn Programming in 21 Days)

Twenty-one days. CUDA C++ is the spine; Python rides beside it the whole way. Every speed claim is measured, not just described.

A CUDA kernel that launches with no error is not one that ran correctly: CUDA errors are asynchronous and silent unless checked by hand. The bottleneck is rarely the arithmetic -- it is the PCIe link.

Day 1 sets a NumPy loop beside a CuPy call, no kernel written yet, to show what a GPU buys and what the transfer costs. By Day 9 memory access is coalesced across a 32-thread warp instead of fought line by line. By Day 15 a naive reduction becomes a three-stage tiled one, benchmarked against numpy.sum. Day 21 closes with a performance checklist and a project brief of your own.

234 figures across 21 chapters, each a real measurement or hardware schematic. 187 listings in CUDA C++, CuPy or Numba, and every printed output captured from a real run on an RTX 4070. Every idea appears in more than one language, so a reader stuck in one can read the other.

YOUR 21-DAY PATH

Day 1 - Why a Different Kind of Chip? Parallel Thinking From the Ground Up: throughput versus latency, and a first measured speedup.
Day 2 - Setting Up, Toolkit, Driver, and Your First Run: nvidia-smi, nvcc, and a device query.
Day 3 - Your First Kernel: __global__, the launch syntax, and SAXPY in CUDA C++, Numba and CuPy.
Day 4 - Threads, Blocks, and Grids: threadIdx, blockDim, the global-index formula, the grid-stride loop.
Day 5 - Vector Addition, End to End: the five steps of every CUDA program, cudaMalloc to a checked result.
Day 6 - Moving Data Between Host and Device: pinned memory, measured bandwidth, unified memory.
Day 7 - When Kernels Go Wrong, Error Checking and Debugging: cudaGetLastError, compute-sanitizer, a bug hunt.
Day 8 - The Memory Hierarchy: global, shared and constant memory, and a measured latency ladder.
Day 9 - Coalesced Memory Access: the 32-thread warp, strided versus contiguous access, array-of-structs.
Day 10 - Shared Memory and Tiling: shared memory as scratchpad, bank conflicts, a tiled matrix transpose.
Day 11 - Synchronization and Race Conditions: __syncthreads, a raced reduction, and why blocks cannot sync.
Day 12 - Warps and Divergence: lockstep execution, branch divergence, and a warp-shuffle reduction.
Day 13 - Occupancy and Launch Configuration: registers, shared memory, and the block-size sweet spot.
Day 14 - Measuring, Timing, Events, and a First Roofline: CUDA events, Nsight, the roofline model.
Day 15 - Reduction, the First Hard Kernel: tree reduction in shared memory, and a three-stage speedup ladder.
Day 16 - Atomics and a Histogram: atomicAdd, contention, and a privatized histogram.
Day 17 - Scan, Prefix Sum: inclusive and exclusive scan, and the Blelloch up-sweep, down-sweep.
Day 18 - Streams and Overlap: cudaMemcpyAsync, pinned memory, and a double-buffered pipeline.
Day 19 - Standing on Libraries, Thrust, cuBLAS, cuFFT, and CuPy: why a library beats a hand-rolled kernel.
Day 20 - A Real Application, Image Convolution on the GPU: a tiled kernel, verified against SciPy.
Day 21 - Wrapping Up and Where to Go Next: a performance checklist, the measure-first mindset, a project brief.

WHO IT IS FOR. A programmer who knows a little C or C++ and a little Python, and wants to write, debug and profile a GPU kernel rather than just call one. You need a C/C++ compiler and Python with NumPy; a local GPU helps but is not required, since CuPy and Numba carry the ideas on a cloud notebook. No prior parallel programming, and no framework beyond the CUDA Toolkit.

Download book source code from: {GitHub}/{mohammadnsaqib}

436 pages, black and white.