Two widely followed surveys “significantly overestimated” how many American adults got their first dose of a Covid-19 vaccine last spring, says an article in the scientific journal Nature that was published online on Wednesday.
The surveys have large numbers of participants but nevertheless aren’t representative of the US adult population, the article says. This flaw is a symptom of a bigger problem, which is the belief that asking lots of people to respond to a survey can make up for deficiencies in its design. It’s what one of the Nature authors, the statistician Xiao-Li Meng, calls the big data paradox. “The more the data, the surer we fool ourselves,” he wrote in 2018 in The Annals of Applied Statistics.
The Nature article analyzes two surveys of Covid-19 vaccine uptake: One is the Delphi-Facebook survey, which gathers about 250,000 responses a week and which its backers describe as unmatched in “detail and scale” for a public health emergency. The other is the Census Bureau’s Household Pulse Survey, which gathers about 75,000 responses every two weeks. The article says that earlier this year the Delphi-Facebook survey overestimated the uptake of Covid-19 vaccines by 17 percentage points (70 percent versus the actual 53 percent), while the Census Bureau’s Household Pulse Survey overestimated the uptake by about 14 percentage points.
Those are big misses, far bigger than the surveys’ own estimates of their margins of error. The reason we know the surveys were wrong is that we can compare their estimates for Jan. 9 to May 19 of this year with verified records of vaccinations for that period that were published on May 26 by the Centers for Disease Control and Prevention.
A much smaller survey conducted by Axios and Ipsos was off by only 4 percentage points, the Nature article says. That survey is based on an online panel with only about 1,000 responses per week, but it uses best practices for obtaining a representative cross-section.
By one measure the Delphi-Facebook survey produced results that are no better than a simple random sample of just 10 people, the authors of the Nature article write. “Our central message,” they conclude, “is that data quality matters more than data quantity, and that compensating the former with the latter is a mathematically provable losing proposition.”
A team of six scholars from Harvard University and the University of Oxford did the research. They are a mix of statisticians, political scientists and computer scientists. The lead authors are Valerie Bradley of Oxford and Shiro Kuriwaki, formerly of Harvard, now an incoming assistant professor at Yale. The senior team members are Seth Flaxman of Oxford and Meng of Harvard. The other two authors are Michael Isakov of Harvard and Dino Sejdinovic of Oxford.
Meng has been calling attention to the big data paradox for years. I wrote about his work earlier this year when I was working for Bloomberg Businessweek. The intuition is that if you solicit opinions about Taylor Swift while you’re at one of her concerts, you won’t be getting a good read on overall opinion. As I wrote:
In a perfectly random sample there’s no correlation between someone’s opinion and their chance of being included in the data. If there’s even a 0.5 percent correlation — i.e., a small amount of selection bias — the non-random sample of 2.3 million will be no better than the random sample of 400, Meng says.
That’s a reduction in effective sample size of 99.98 percent.
That’s not just theory: Statisticians estimate that there was a 0.5 percent correlation contaminating the 2016 presidential polls, presumably because supporters of Donald Trump were slightly less likely to express their preference to pollsters. That’s why so many pollsters were caught by surprise when Trump won. The 2020 polls suffered similar problems.
Both the Delphi-Facebook survey and the Census Bureau’s Household Pulse Survey have been widely quoted in the news and cited in academic research. The Delphi Research Group, a research team at Carnegie Mellon University that collaborates with Facebook, has a web page with links to 15 publications, including ones in journals such as Science and Lancet Digital Health. (There are ways for scholars to make valid use of imperfect data, such as re-weighting it to better approximate the makeup of the population, but they need to tread carefully, the authors say.)
I reached out to the Delphi-Facebook and Census Bureau teams for their responses to the Nature article. A Census Bureau spokeswoman directed me to the bureau’s description of the Household Pulse Survey, which says that it “is designed to deploy quickly and efficiently” but is experimental. The page adds, “Census Bureau experimental data may not meet all of our quality standards.”
The Delphi-Facebook team argues in a written response that its survey “in fact performs well for its intended goals.” Alex Reinhart, an assistant teaching professor of statistics and data science at Carnegie Mellon who is the university’s principal investigator on the survey, said in an interview that it was put together at the request of the C.D.C. to meet an urgent need for data early in the pandemic. He said it was pretty good at detecting changes in infection rates and vaccination rates in local areas, but not so good at getting the levels right. (Although lately the problem has been different: Delphi-Facebook surveys have shown little change in self-reported vaccination rates even though actual vaccination rates have risen.)
“Official data reporting has gotten a lot better since early 2020,” Reinhart said. “And some of the goals have changed. Forecasting seems less important. It’s not obvious exactly what we should focus on next as the pandemic winds down. Maybe decrease the size of the survey, focus it more narrowly or discontinue it.”
I also exchanged emails with the public health scholars Phoenix Do of the University of Wisconsin-Milwaukee and Reanne Frank of Ohio State University, who have used the Delphi-Facebook data in their work. They wrote that they “adjusted for metropolitan status, political affiliation and education — factors that were known to be nonrepresentative in the Delphi-Facebook data.” They added that “there is currently a dearth of data sources” on socioeconomic factors related to Covid-19: “The Delphi-Facebook data has that needed information; moreover, it has been released in near real-time, enabling researchers such as ourselves to investigate these types of questions in a timely manner.”
The New York Times
