NVIDIA GPU Programming in CUDA and Python in 14 Days

Fourteen days of drills on GPU programming, written in Python and measured on a real NVIDIA card.

This is a drill manual, not a narrative course. Every one of the 463 listings was executed on a real GPU, and every printed output block and every printed timing is a paste from that run — including the ones that came out level, and the ones that contradicted the theory. Each chapter states a rule in as few words as it can, shows the rule working in a short numbered example with its captured output, and hands it straight back as problems. The programming problems build one program across the whole book: gridsim, a grid relaxation that starts as a field of floats on the device in Chapter 1 and ends as a tuned simulation with a shared-memory stencil, a reduced convergence measure and a measured launch configuration.

Most kernels here are written in Python with Numba's @cuda.jit. Where the C level is the lesson, the kernel is written in CUDA C and compiled at run time through CuPy, so a reader finishes able to read and write both and to move arrays between them without a copy.

The fourteen chapters carry the whole working core. The first week covers the execution model and the first kernel, thread indexing in one and two dimensions, device memory and transfers, the CuPy array layer, honest timing and device properties, warps and divergence, and global memory access patterns. The second week covers shared memory and bank conflicts, synchronization and atomics, reduction and prefix sum, writing CUDA C kernels from Python, tiled matrix multiplication, streams and overlap, and occupancy and launch tuning.

The measurements are reported as they came out. A tiled matrix multiply is 2.6 times the naive one and the library is still eight times faster than both. A pipeline in eight streams beats one stream by three per cent, not the forty the arithmetic promises. Raising occupancy by capping registers makes one kernel slower. A book that only printed the wins would be easier to write and worth less to work through.

Your 14-Day Path

• 1. THE GPU AND THE FIRST KERNEL



• 2. THREAD INDEXING



• 3. DEVICE MEMORY AND TRANSFERS



• 4. ARRAY PROGRAMMING WITH CUPY



• 5. TIMING, ERRORS, AND DEVICE PROPERTIES



• 6. WARPS AND DIVERGENCE



• 7. GLOBAL MEMORY ACCESS PATTERNS



• 8. SHARED MEMORY



• 9. SYNCHRONIZATION AND ATOMICS



• 10. REDUCTION AND SCAN



• 11. WRITING CUDA C KERNELS FROM PYTHON



• 12. MATRIX MULTIPLICATION AND TILING



• 13. STREAMS, EVENTS, AND OVERLAP



• 14. OCCUPANCY AND LAUNCH TUNING

Who It's For

Students on a parallel-computing or high-performance-computing course. Readers preparing for an examination or a technical interview. Working scientists and engineers whose NumPy code has outgrown the CPU. Python programmers who want the GPU without first learning C++. No prior CUDA is assumed; comfort with Python and NumPy is.

336 worked examples. 448 figures. 672 numbered problems. 380 pages. Measured on an RTX 4070 with CuPy 14.1 and Numba 0.67.

You need an NVIDIA GPU and about an hour a day. Start Chapter 1.