Getting numbers is easy; getting numbers you can trust is hard. This practical guide by experimentation leaders at Google, LinkedIn, and Microsoft will teach you how to accelerate innovation using trustworthy online controlled experiments, or A/B tests.
"A/B testing is the gold standard of creating verifiable and repeatable experiments, and this book is its definitive text" -- Steve Blank, father of modern entrepreneurship, author of The Startup Owner's Manual and The Four Steps to the Epiphany
"This book is a great resource for executives, leaders, researchers or engineers looking to use online controlled experiments" -- Harry Shum, Executive Vice President, Microsoft Artificial Intelligence and Research Group
"A great book that is both
rigorous and accessible. Readers will learn how to bring trustworthy controlled experiments, which have
revolutionized internet product development, to their organizations" --
Adam D'Angelo, Co-founder and CEO of Quora and prior CTO of Facebook
"Kohavi, Tang and Xu have a wealth of experience and excellent advice to convey, so the book has lots of practical real world examples and lessons learned over many years of the application of these techniques at scale." -- Jeff Dean, Google Senior Fellow, and SVP, Google Research
"The secret sauce for a successful online business is experimentation. But it is a secret no longer. Here three masters of the art describe the ABCs of A/B testing so that you too can continuously improve your online services." -- Hal Varian, Chief Economist, Google, and author of Intermediate Microeconomics: A Modern Approach
"This is the new bible of how to get from data to decisions in the digital age." -- Scott Cook, Intuit Co-founder & Chairman of the executive committee
Based on practical experiences at companies that each run more than 20,000 controlled experiments a year, the authors share examples, pitfalls, and advice for students and industry professionals getting started with experiments, plus deeper dives into advanced topics for practitioners who want to improve the way they make data-driven decisions. Learn how to
- Use the scientific method to evaluate hypotheses using controlled experiments.
- Define key metrics and ideally an Overall Evaluation Criterion.
- Test for trustworthiness of the results and alert experimentersto violated assumptions.
- Build a scalable platform that lowers the marginal cost of experiments close to zero.
- Avoid pitfalls like carryover effects and Twyman's law * Understand how statistical issues play out in practice.
Ron Kohavi is a VP and Technical Fellow at Airbnb. He was previously a Technical Fellow and Corporate VP at Microsoft. Prior to Microsoft, he was the director of data mining and personalization at Amazon.com. He has a PhD in Computer Science for Stanford University. His papers have over 40,000 citations and three of his papers are in the top 1,000 most-cited papers in Computer Science.
Diane Tang is a Google Fellow, with expertise in large-scale data analysis and infrastructure, online controlled experiments, and ads systems. She has an AB from Harvard and MS/PhD from Stanford, and has patents and publications in mobile networking, information visualization, experiment methodology, data infrastructure, and data mining / large data.
Ya Xu heads Data Science and Experimentation at LinkedIn. She has led LinkedIn to become one of the most well-regarded companies when it comes to A/B testing. Before LinkedIn, she worked at Microsoft and received a PhD in Statistics from Stanford University. She is widely regarded as one of the premier scientists, practitioners and thought leaders in the domain of experimentation, with several filed patents and publications. She is also a frequent speaker at top conferences, universities and companies across the country.
