Deep Learning for Computer Architects (Synthesis Lectures on Computer Architecture)

This is a primer written for computer architects in the new and rapidly evolving field of deep learning. It reviews how machine learning has evolved since its inception in the 1960s and tracks the key developments leading up to the emergence of the powerful deep learning techniques that emerged in the last decade.

Machine learning, and specifically deep learning, has been hugely disruptive in many fields of computer science. The success of deep learning techniques in solving notoriously difficult classification and regression problems has resulted in their rapid adoption in solving real-world problems. The emergence of deep learning is widely attributed to a virtuous cycle whereby fundamental advancements in training deeper models were enabled by the availability of massive datasets and high-performance computer hardware.

It also reviews representative workloads, including the most commonly used datasets and seminal networks across a variety of domains. In addition to discussing the workloads themselves, it also details the most popular deep learning tools and show how aspiring practitioners can use the tools with the workloads to characterize and optimize DNNs.

The remainder of the book is dedicated to the design and optimization of hardware and architectures for machine learning. As high-performance hardware was so instrumental in the success of machine learning becoming a practical solution, this chapter recounts a variety of optimizations proposed recently to further improve future designs. Finally, it presents a review of recent research published in the area as well as a taxonomy to help readers understand how various contributions fall in context.

Brandon Reagen is a Ph.D. candidate at Harvard University. He received his B.S. degree in Computer Systems Engineering and Applied Mathematics from University of Massachusetts, Amherst in 2012 and his M.S. in Computer Science from Harvard in 2014. His research spans the fields of Computer Architecture, VLSI, and Machine Learning with specific interest in designing extremely efficient hardware to enable ubiquitous deployment of Machine Learning models across all compute platforms.

Robert Adolf is a Ph.D. candidate in computer architecture at Harvard University. After earning a B.S. in Computer Science from Northwestern University in 2005, he spent four years doing benchmarking and performance analysis of supercomputers at the Department of Defense. In 2009, he joined Pacific Northwest National Laboratory as a research scientist, where he lead a team building large-scale graph analytics on massively multithreaded architectures. His research interests revolve around modeling, analysis, and optimization techniques for high-performance software, with a current focus on deep learning algorithms. His philosophy is that the combination of statistical methods, code analysis, and domain knowledge leads to better tools for understanding and building fast systems.

Paul Whatmough leads research on computer architecture for Machine Learning at ARM Research, Boston, MA. He is also an Associate in the School of Engineering and Applied Science at Harvard University. Dr. Whatmough received the B.Eng. degree (with first class Honors) from the University of Lancaster, U.K., M.Sc. degree (with distinction) from the University of Bristol, U.K., and Doctorate degree from University College London, U.K. His research interests span algorithms, computer architecture, and circuits. He has previously led various projects on hardware accelerators, Machine Learning, SoC architecture, Digital Signal Processing (DSP), variation tolerance, and supply voltage noise.

Gu-Yeon Wei is Gordon McKay Professor of Electrical Engineering and Computer Science in the School of Engineering and Applied Sciences (SEAS) at Harvard University. He received his B.S., M.S., and Ph.D. degrees in Electrical Engineering from Stanford University in 1994, 1997, and 2001, respectively. His research interests span multiple layers of a computing system: mixed-signal integrated circuits, computer architecture, and design tools for efficient hardware. His research efforts focus on identifying synergistic opportunities across these layers to develop energy-efficient solutions for a broad range of systems from flapping-wing microrobots to machine learning hardware for IoT/edge devices to specialized accelerators for large-scale servers.

David Brooks is the Haley Family Professor of Computer Science in the School of Engineering and Applied Sciences at Harvard University. Prior to joining Harvard, he was a research staff member at IBM T. J. Watson Research Center. Prof. Brooks received his B.S. in Electrical Engineering at the University of Southern California and M.A. and Ph.D. degrees in Electrical Engineering at Princeton University. His research interests include resilient and power-efficient computer hardware and software design for high-performance and embedded systems. Prof. Brooks is a Fellow of the IEEE and has received several honors and awards including the ACM Maurice Wilkes Award, ISCA Influential Paper Award, NSF CAREER award, IBM Faculty Partnership Award, and DARPA Young Faculty Award.