LLM & Transformer Interview Essentials A?Z: A Silicon Valley Insider's Guide

Ever since the release of ChatGPT in November 2022, the advances in AI and its growth in the public consciousness have been tremendous. At the heart of these epochal changes lies a single machine learning component: the transformer. Like the transistor of the 1960s, the transformer of the 2020s has given us an efficient, composable structure for solving generalized problems, and the transformer can be replicated, scaled up, modularized, and miniaturized however we might need. While the large language models (LLMs) that underpin products like ChatGPT are the most popular, these are but one configuration of the transformer.

This book is written for the engineers, machine learning scientists, data scientists, and technologists who are either working with LLMs or transformers, or who are currently trying to break into the field. This technology is so new that machine learning interviews for such positions have not yet been standardized and commoditized to the level of LeetCode, so a broad familiarity with the core concepts is required. Indeed, it is possible to be an expert on one aspect of the LLM space yet still be blindsided by a comparatively rudimentary question on another.

Table of Contents (in alphabetical order)
A ⇒ Attention
B ⇒ BitNet
C ⇒ Chinchilla Scaling Laws
D ⇒ Distillation
E ⇒ Embeddings
F ⇒ FlashAttention
G ⇒ GPTQ
H ⇒ H100
I ⇒ InstructGPT
K ⇒ KV Cache
L ⇒ LoRA
M ⇒ Mixture of Experts
N ⇒ NCCL
P ⇒ Pipeline Parallelism
Q ⇒ Quantization
R ⇒ RoPE
S ⇒ Structured Sparsity
T ⇒ Tensor Parallelism
V ⇒ Vanilla Transformer
W ⇒ WxAyKVz
Z ⇒ ZeRO