Standard Deviations: Flawed Assumptions, Tortured Data, and Other Ways to Lie with Statistics - Hardcover

Smith, Gary

Hardcover

ISBN 10: 146830920X ISBN 13: 9781468309201

Publisher: Abrams Press, 2014

Synopsis

Or that Asian Americans are most susceptible to heart attacks on the fourth day of the month? Or that drinking a full pot of coffee every morning will add years to your life, but one cup a day increases the risk of pancreatic cancer? All of these “facts” have been argued with a straight face by credentialed researchers and backed up with reams of data and convincing statistics. As Nobel Prize–winning economist Ronald Coase once cynically observed, “If you torture data long enough, it will confess.” Lying with statistics is a time-honored con. In Standard Deviations, economics professor Gary Smith walks us through the various tricks and traps that people use to back up their own crackpot theories. Sometimes, the unscrupulous deliberately try to mislead us. Other times, the well-intentioned are blissfully unaware of the mischief they are committing. Today, data is so plentiful that researchers spend precious little time distinguishing between good, meaningful indicators and total rubbish. Not only do others use data to fool us, we fool ourselves. With the breakout success of Nate Silver’s The Signal and the Noise, the once humdrum subject of statistics has never been hotter. Drawing on breakthrough research in behavioral economics by luminaries like Daniel Kahneman and Dan Ariely and taking to task some of the conclusions of Freakonomics author Steven D. Levitt, Standard Deviations demystifies the science behind statistics and makes it easy to spot the fraud all around.

About this title

About the Author:

Gary Smith is the Fletcher Jones Professor of Economics at Pomona College in Claremont, California. He received his Ph.D. in Economics from Yale University and taught there as Assistant Professor for seven years. He has won two teaching awards and authored more than seventy academic papers, nine textbooks, and seven educational software programs. This is his first trade book.

Excerpt. © Reprinted by permission. All rights reserved.:

DID YOU KNOW that building a quarry in your backyard can increase your property value? Or that America’s unemployment rate is actually zero? Or that drinking a full pot of coffee every morning adds years to your life, but drinking two cups a day increases the risk of cancer?

If you believe the above, then Professor Gary Smith has a World Cup–predicting octopus he’d like to show you. The sad truth is that “facts” like these are routinely presented with a straight face by credentialed academics and backed up with reams of raw data. In Standard Deviations, Smith skillfully unpacks the various ways we are duped by data every day. He deftly demonstrates how a straightforward set of findings can be teased and manipulated to reflect whatever the researcher wants to see.

Lying with statistics is a time-honored con, and in this age of Big Data even the most accredited findings can be suspect. Blending the keen statistical eye of Nate Silver with the probing insights of Daniel Kahneman and Dan Ariely, Smith demystifies the math behind the dismal science, making it easy to spot flaws all around and find the truth hidden in plain sight.

MORE ADVANCE PRAISE FOR STANDARD DEVIATIONS

“Gary Smith’s Standard Deviations is both a statement of principles for doing statistical inference correctly and a practical guide for interpreting the (supposedly) data-based inferences other people have drawn. Cleverly written and engaging to read, the book is full of concrete examples that make clear not just what Smith is saying but why it matters. Readers will discover that lots of what they thought they’d learned is wrong, and they’ll understand why.”

—BENJAMIN M. FRIEDMAN

William Joseph Maier Professor
of Political Economy, Harvard University

“Standard Deviations shows in compelling fashion why humans are so susceptible to the misuse of statistical evidence and why this matters. I know of no other book that explains important concepts such as selection bias in such an entertaining and memorable manner.”

—RICHARD J. MURNANE

Thompson Professor of Education and Society,
Harvard Graduate School of Education

“We all learn in school that there are three kinds of lies: lies, damn lies, and statistics. Gary Smith’s new book imparts true substance to this point by setting forth myriad examples of how and why statistics and data-crunching at large are susceptible to corruption. The great risk today is that the young will forget that deductive logic is vastly more powerful than inductive logic.”

—HORACE “WOODY” BROCK

President, Strategic Economic Decisions, Inc.

“Statistical reasoning is the most used and abused form of rhetoric in the field of finance. Standard Deviations is an approachable and effective means to arm oneself against the onslaught statistical hyperbole in our modern age. Professor Smith has done us all a tremendous service.”

—BRYAN WHITE

Managing Director, BlackRock, Inc.

“It’s entertaining, it’s gossipy, it’s insightful—and it’s destined to be a classic. Based on a lifetime of experience unraveling the methodical blunders that remain all too frequent, this book communicates Gary Smith’s wisdom about how not to do a data analysis. Smith’s engaging rendering of countless painful mistakes will help readers avoid the pitfalls far better than merely mastering theorems.”

—EDWARD E. LEAMER

Distinguished Professor and Chauncey J. Medberry
Chair in Management, UCLA

“Standard Deviations will teach you how not to be deceived by lies masquerading as statistics. Written in an entertaining style with contemporary examples, this book should appeal to everyone, whether interested in marriages or mortgages, the wealth of your family, or the health of the economy. This should be required reading for everyone living in this age of (too much?) information.”

—ARTHUR BENJAMIN

Professor of Mathematics, Harvey Mudd College
and author of Secrets of Mental Math

“One of those rare books that make people better for having read it.”

—JAY CORDES

Senior Manager, RookMedia.net

“Most of the authoritative, sciencey-sounding claims we’re fed by the media are polluted by distortions, biases, and plain old errors. In Standard Deviations, Gary Smith sets the record straight.”

—DAVID H. FREEDMAN

Author of Wrong: Why Experts Keep Failing Us—and
How to Know When Not to Trust Them

Copyright

To my wife Margaret and my children Josh,
Jo, Chaska, Cory, Cameron, and Claire

INTRODUCTION

WE LIVE IN THE AGE OF BIG DATA. THE POTENT COMBINATION of fast computers and worldwide connectivity is continually praised—even worshipped. Over and over, we are told that government, business, finance, medicine, law, and our daily lives are being revolutionized by a newfound ability to sift through reams of data and discover the truth. We can make wise decisions because powerful computers have looked at the data and seen the light.

Maybe. Or maybe not. Sometimes these omnipresent data and magnificent computers lead to some pretty outlandish discoveries. Case in point, serious people have seriously claimed that:

·   Messy rooms make people racist.

·   Unborn chicken embryos can influence computer random-event generators.

·   When the ratio of government debt to GDP goes above 90 percent, nations nearly always slip into recession.

·   As much as 50 percent of the drop in the crime rate in the United States over the past twenty years is because of legalized abortion.

·   Drinking two cups of coffee a day substantially increases the risk of pancreatic cancer.

·   The most successful companies tend to become less successful, while the least successful companies tend to become more successful, so that soon all will be mediocre.

·   Athletes who appear on the cover of Sports Illustrated or Madden NFL are jinxed in that they are likely to be less successful or injured.

·   Living near power lines causes cancer in children.

·   Humans have the power to postpone death until after important ceremonial occasions.

·   Asian Americans are more susceptible to heart attacks on the fourth day of the month.

·   People live three to five years longer if they have positive initials, like ACE.

·   Baseball players whose first names began with the letter D die, on average, two years younger than players whose first names began with the letters E through Z.

·   The terminally ill can be cured by positive mental energy sent from thousands of miles away.

·   When an NFC team wins the Super Bowl, the stock market almost always goes up.

·   You can beat the stock market by buying the Dow Jones stock with the highest dividend yield and the second lowest price per share.

These claims—and hundreds more like them—appear in newspapers and magazines every day even though they are surely false. In today’s Information Age, our beliefs and actions are guided by torrents of meaningless data. It is not hard to see why we repeatedly draw false inferences and make bad decisions. Even if we are reasonably well informed, we are not always alert to the ways in which data are biased or irrelevant, or to the ways in which scientific research is flawed or misleading. We tend to assume that computers are infallible—that no matter what kind of garbage we put in, computers will spit out gospel. It happens not just to laymen in their daily lives, but in serious research by diligent professionals. We see it in the popular press, on television, on the Internet, in political campaigns, in academic journals, in business meetings, in courtrooms, and, of course, in government hearings.

Decades ago, when data were scarce and computers nonexistent, researchers worked hard to gather good data and thought carefully before spending hours, even days, on painstaking calculations. Now with data so plentiful, researchers often spend too little time distinguishing between good data and rubbish, between sound analysis and junk science. And, worst of all, we are too quick to assume that churning through mountains of data can’t ever go wrong. We rush to make decisions based on the balderdash these machines dish out—to increase taxes in the midst of a recession, to trust our life savings to financial quants who impress us because we don’t understand them, to base business decisions on the latest management fad, to endanger our health with medical quackery, and—worst of all—to give up coffee.

Ronald Coase cynically observed that, “If you torture the data long enough, it will confess.” Standard Deviations is an exploration of dozens of examples of tortuous assertions that, with even a moment’s reflection, don’t pass the smell test. Sometimes, the unscrupulous deliberately try to mislead us. Other times, the well-intentioned are blissfully unaware of the mischief they are committing. My intention in writing this book is to help protect us from errors—both external and self-inflicted. You will learn simple guidelines for recognizing bull when you see it—or say it. Not only do others use data to fool us, we often fool ourselves.

1

PATTERNS, PATTERNS, PATTERNS

YOUTH SOCCER IS A VERY BIG DEAL WHERE I LIVE IN SOUTHERN California. It’s a fun, inexpensive sport that can be played by boys and girls of all sizes and shapes. I initially didn’t know anything about soccer. All I knew was that, every weekend, the city parks and school grounds were filled with kids in brightly colored uniforms chasing soccer balls while their parents cheered. When my son was old enough, we were in.

By the time the 2010 World Cup came around, my son was playing on one of the top soccer teams in Southern California. I was the manager and a fanatic about soccer, so naturally he and I watched every World Cup match we could. The opponents in the 2010 championship game were Netherlands and Spain, two extraordinarily talented teams from underachieving nations that often disappointed their supporters. Which country would finally win the World Cup? I loved the Dutch, who had won all six of their World Cup games, scoring twelve goals while allowing only five, and had knocked out the mighty Brazil and Uruguay. But then I heard about Paul the octopus, who had correctly predicted the winners of seven World Cup games by choosing food from plastic boxes with the nations’ flags on them. Paul the Oracle had picked Spain, and the world now seemed certain of a Spanish victory.

What the heck was going on? How could a slimy, pea-brained invertebrate know more about soccer than I did? I laughed and waited for Paul the Omniscient to get his comeuppance. Except he didn’t. The Dutch did not play with their usual creativity and flair. In a brutal, cynical match, with fourteen yellow cards—nine given to the dirty Dutchmen—Spain scored the winning goal with four minutes left in the game.

How could an octopus living in a tank have predicted any of this? Had Paul ever seen a soccer game? Did Paul even have a brain?

It turns out that octopuses are among the most intelligent invertebrates, but that isn’t saying much—sort of like being the world’s tallest midget. Still, Paul made eight World Cup predictions and got every single one right. Not only that, Paul made six predictions during the 2008 European Football Championships and got four right. Overall, that’s twelve out of fourteen correct, which in the eyes of many would be considered statistical proof of Paul’s psychic abilities. But were there really enough data?

If a fair coin is flipped fourteen times, the chances of twelve or more heads are less than one percent. In the same way, if Paul were just a hapless guesser with a 50 percent chance of making a correct prediction, the probability that he would make so many correct predictions is less than 1 percent, a probability so low that it is considered “statistically significant.” The chances of Paul being correct so many times are so small that, logically, we can rule out luck as an explanation. With his consistency, Paul had demonstrated that he was not merely a lucky guesser. He was truly Paul the Psychic Octopus!

And yet, something didn’t seem quite right. Is it really possible for an octopus to predict the future? Paul’s performance raises several issues that are endemic in statistical studies. Paul was not a psychic (surprise, surprise), but he is a warning of things to watch out for the next time you hear some fanciful claim.

CONFOUNDING EFFECTS

First, let’s look at how Paul made his predictions. At feeding time, he was shown two clear plastic boxes with the national flags of the opposing teams glued to the front of the boxes. The boxes contained identical yummy treats, such as a mussel or an oyster. Whichever box Paul opened first was the predicted winner.

Octopuses don’t know much about soccer, but they do have excellent eyesight and good memories. One time, an octopus at the New England Aquarium decided he didn’t like a volunteer and shot salt water at her whenever he saw her. She left the aquarium to go to college, but when she returned months later, the octopus remembered her and immediately drenched her with salt water again. In an experiment at a Seattle aquarium, one volunteer fed the octopuses while another wearing identical clothes irritated the octopuses with a stick. After a week of this, most of the octopuses could tell who was who. When they saw the good person, they moved closer; when they saw the bad person, they moved away (and sometimes shot water at him for good measure).

Paul the Psychic Octopus happened to be living in an aquarium in Germany and, except for the Spain-Netherlands World Cup final, Paul only predicted games involving Germany. In eleven of the thirteen games involving Germany, Paul picked Germany—and Germany won nine of these eleven games. Was Paul picking Germany because he had analyzed their opponents carefully or because he had an affinity for the German flag? Paul was almost certainly color blind, but experiments have shown that octopuses recognize brightness and are attracted to horizontal shapes. Germany’s flag has three vivid horizontal stripes, as do the flags of Serbia and Spain, the only other countries Paul selected. Indeed, the Spanish and German flags are pretty similar, which may explain why Paul picked Spain over Germany in one of the two matches they played and picked Spain over the Netherlands in the World Cup final. The only game in which Paul did not choose the German or Spanish flag was a match between Serbia and Germany.

The flag was apparently a confounding factor in that Paul wasn’t picking the best soccer team. He was choosing his favorite flag. Paul the Omniscient was just a pea-brained octopus after all.

Figure 1.1: Paul’s Favorite Flags

Germany (eleven times)

Spain (twice)

Serbia (once)

SELECTIVE REPORTING AND MISREPORTING

Another explanation for Paul’s success is that too many pe...

About this edition

• Publisher: Abrams Press
• Publication date: 2014
• ISBN 10: 146830920X
• ISBN 13: 9781468309201
• Binding: Hardcover
• Edition number: 1
• Number of pages: 304