Convergence rates of least squares regression estimators with heavy-tailed errors

Qiyang Han, Jon A. Wellner

Research output: Contribution to journalArticlepeer-review

21 Scopus citations

Abstract

We study the performance of the least squares estimator (LSE) in a general nonparametric regression model, when the errors are independent of the covariates but may only have a pth moment (p = 1). In such a heavy-tailed regression setting, we show that if the model satisfies a standard "entropy condition" with exponent a α ϵ (0,2), then the L2 loss of the LSE converges OP n- 1 2+α n-12 + 1 2p. at a rate Such a rate cannot be improved under the entropy condition alone. This rate quantifies both some positive and negative aspects of the LSE in a heavy-tailed regression setting. On the positive side, as long as the errors have p = 1 + 2/a moments, the L2 loss of the LSE converges at the same rate as if the errors are Gaussian. On the negative side, if p < 1 + 2/a, there are (many) hard models at any entropy level a for which the L2 loss of the LSE converges at a strictly slower rate than other robust estimators. The validity of the above rate relies crucially on the independence of the covariates and the errors. In fact, the L2 loss of the LSE can converge arbitrarily slowly when the independence fails. The key technical ingredient is a new multiplier inequality that gives sharp bounds for the "multiplier empirical process" associated with the LSE. We further give an application to the sparse linear regression model with heavytailed covariates and errors to demonstrate the scope of this new inequality.

Original languageEnglish (US)
Pages (from-to)2286-2319
Number of pages34
JournalAnnals of Statistics
Volume47
Issue number4
DOIs
StatePublished - 2019
Externally publishedYes

All Science Journal Classification (ASJC) codes

  • Statistics and Probability
  • Statistics, Probability and Uncertainty

Keywords

  • Heavy-tailed errors
  • Least squares estimation
  • Multiplier empirical process
  • Multiplier inequality
  • Nonparametric regression
  • Sparse linear regression

Fingerprint

Dive into the research topics of 'Convergence rates of least squares regression estimators with heavy-tailed errors'. Together they form a unique fingerprint.

Cite this