Parallel Python with Dask, Second Edition

This fully revised edition teaches you to scale Python across cores, machines, and GPUs using the libraries you already know. I'll teach you to build Dask arrays, dataframes, bags and delayed graphs, and to tune partitions and chunks so the cluster actually earns its keep. I'll also show you to use the Dask expression system to let the optimiser push filters down and prune columns before a byte moves.

As you'll see, you'll be working with pandas 3.0 throughout, including copy-on-write, PyArrow-backed strings and the new expression API. There are dedicated chapters that take you through terabyte-scale multi-dimensional data in HDF5, NetCDF, TIFF and Zarr, with Xarray, Kerchunk and cloud-native datacubes. The good thing about Dask is that it's the lighter, and the quicker choice than Spark in most scenarios. After that, we'll move on to GPU acceleration with RAPIDS and cuDF, distributed XGBoost and dask-ml for scaled machine learning, and production deployment on Kubernetes with monitoring, security, and cost control. Every chapter builds on one ongoing project, so the skills you learn are added to over time instead of being spread out.

Key Learnings

• Diagnose whether a workload needs parallelism, a bigger machine, or better code.
• Size partitions and array chunks so clusters stop thrashing and start scaling.
• Read task graphs and dashboard panels to locate bottlenecks within minutes.
• Exploit the Dask expression system for predicate pushdown and column projection.
• Migrate pandas code to 3.0 copy-on-write and PyArrow-backed string dtypes.
• Stream terabyte HDF5, NetCDF, TIFF, and Zarr archives without exhausting memory.
• Build cloud-native, versioned datacubes using Zarr v3, Kerchunk, and Icechunk.
• Choose between Dask, Spark, Ray, and Polars using measured, honest criteria.
• Accelerate dataframes and gradient boosting on GPUs with RAPIDS and dask-cuda.
• Deploy, secure, monitor, and cost-control Dask clusters on production Kubernetes.

Table of Content

1. Up and Running with Dask
2. Working with Dask Collections
3. Tuning Partitions and Chunks
4. Optimizing Queries with Expression System
5. Scaling Out with Distributed Clusters
6. Scaling pandas 3.0 with Dask
7. Reading and Writing Data at Scale
8. Working with Labeled Arrays in Xarray
9. Reading HDF5 and NetCDF Archives
10. Scaling Imagery with TIFF
11. Building Cloud-native Datacubes with Zarr
12. Accelerating Dask with GPUs
13. Operating Dask in Production

Target Audience

If you're wondering whether your Pandas script is stalling and you've tried adding memory but only got a few weeks out of it, then this latest edition on running parallel operations should help you figure out what to do next. You just need to have a basic understanding of Python programming. That's all you need to get reading this book!