Similarities and Differences in Tobacco Control Research Findings from Convenience and Probability Samples

Background Online convenience samples are a quick and low-cost way to study health behavior, but the comparability to findings from probability samples is not yet well understood. Purpose We sought to compare convenience and probability samples' findings for experiments, correlates, and prevalence in the context of tobacco control research. Methods Participants were a probability sample of 5,014 U.S. adults recruited by phone from September 2014 through May 2015 (cost ∼U.S.$620,000) and an online convenience sample of 4,137 U.S. adults recruited through Amazon Mechanical Turk (MTurk) in December 2014 (cost ∼U.S.$17,000). Participants completed a survey with experiments, measures of tobacco product use and demographic characteristics. Results MTurk convenience and probability samples showed the same pattern of statistical significance and direction in almost all experiments (21 of 24 analyses did not differ) and observational studies (19 of 25 associations did not differ). Demographic characteristics of the samples differed substantially (1 of 17 estimates did not differ), with the convenience sample being younger, having more years of education, and including more Whites and Asians. Tobacco product use also differed substantially (1 of 22 prevalence estimates did not differ), with the convenience sample reporting more cigarette and e-cigarette use (median error 19%). Conclusions Using MTurk convenience samples can yield generalizable findings for experiments and observational studies. Prevalence estimates from MTurk convenience samples are likely to be over- or underestimates.