Causal inference and the design and analysis of experiments

Introduction This chapter introduces key concepts in causal inference, with a focus on the design and analysis of experiments. It is principally aimed at readers who do not have formal training or extensive experience in experimentation but who want to get the most out of the chapters in this volume, critically interpret experimental studies and their findings, or understand how experimental methods relate to more conventional, non-experimental methods of public management research. This chapter also serves as a starting place for those interested in designing and carrying out their own experiments and should also be of interest to those reviewing experiments for publication or commissioning experimental research, for example, to inform policy or practice. This chapter begins with a discussion of what counts as a research method producing valid findings about causation and uses this perspective to review conventional, non-experimental, observational methods. Research using observation of correlations to establish causation has limitations as a method, including the ‘research as regression’ approach that has come to dominate public management research. Next, this chapter sets out the fundamental concepts, logic, and advantages of experimentation. It discusses how experiments involve researchers intervening in the world by giving treatments to people or other experimental units. In the experiments that form the focus of this book, treatments are randomly assigned to experimental units placed in groups on the basis of treatment received (e.g., a treatment or control group). Researchers then compare the outcomes between these groups, typically to estimate average treatment effect. This chapter then introduces the counterfactual approach to causation and the potential outcomes framework, which helps further demonstrate the benefits of experimental as opposed to non-experimental methods. Having set out the general approach of experimentation, the main designs of experiment are introduced, including the use of multiple treatments and different timings of the measurement of outcomes. Next, the analysis of data from experiments is discussed. This chapter moves on to examine potential problems with the implementation of experiments that can threaten the validity of findings. It concludes by noting the importance of carrying out a series of experiments in support of research on a topic area and introduces field, survey, and laboratory experiments as types of experiment that are covered in subsequent chapters.