Estimation and Inference with High-Dimensional Data

High-dimensional data are collected in a wide range of disciplines, from biology and medical research, natural sciences, and engineering to social sciences, economics, and finance. Statistical inference with such data has become increasingly important. The objective of this research project is to develop improved statistical methods, algorithms, and theory for estimation and inference with high-dimensional data. The project will implement the new methods in several applications that demonstrate their feasibility, effectiveness, and usefulness. The project will also carry out comprehensive numerical experiments to verify the computational efficiency of the new algorithms and to prove the relevance of the related theory in realistic settings. The numerical work aims to produce concrete evidence of the utility of the approach in wide contexts. The work will foster collaborations between researchers with different expertise, allow students and young researchers to align quickly with cutting edge research, and encourage them to embark on a host of exciting research topics. Special efforts will be devoted to recruiting and encouraging students from underrepresented groups. Software and other tools will be made available to the public, enhancing scientific and data-driven decision making in practical applications.High-dimensional data is an intense area of research in statistics due to its central role in the development and theoretical understanding of some of the most widely used statistical methods in modern time. This research project intends to establish a solid foundation for future work in the emerging topic arising from the convergence of differential-based statistical inference methods and approximate message passing, and their connection to empirical Bayesian methods. It aims to develop the central limit theorem for Stein's unbiased risk estimate, new methods and theory for regularized estimation, new methods and theory for de-biased statistical inference including confidence intervals and regions, and empirical Bayes methods in approximate massage passing. The project intends to produce a rich collection of new tools for such statistical inference, study the theoretical and empirical properties of the newly developed methods, and set the scene for their application in important fields including sociology, economics, neural imaging, signal processing, communications, social networks, bioinformatics, and text analysis. The project findings are expected to have impact as well in other fields of statistics, including causal inference, missing data, survival analysis, compressed sensing, information retrieval, and signal processing, significantly advancing statistics and data science research in general.This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.