TY - JOUR

T1 - Convergence rates of least squares regression estimators with heavy-tailed errors

AU - Han, Qiyang

AU - Wellner, Jon A.

N1 - Funding Information:
Received February 2018; revised May 2018. 1Supported in part by NSF Grant DMS-1566514. MSC2010 subject classifications. Primary 60E15; secondary 62G05. Key words and phrases. Multiplier empirical process, multiplier inequality, nonparametric regression, least squares estimation, sparse linear regression, heavy-tailed errors.
Publisher Copyright:
© 2019 Institute of Mathematical Statistics.

PY - 2019

Y1 - 2019

N2 - We study the performance of the least squares estimator (LSE) in a general nonparametric regression model, when the errors are independent of the covariates but may only have a pth moment (p = 1). In such a heavy-tailed regression setting, we show that if the model satisfies a standard "entropy condition" with exponent a α ϵ (0,2), then the L2 loss of the LSE converges OP n- 1 2+α n-12 + 1 2p. at a rate Such a rate cannot be improved under the entropy condition alone. This rate quantifies both some positive and negative aspects of the LSE in a heavy-tailed regression setting. On the positive side, as long as the errors have p = 1 + 2/a moments, the L2 loss of the LSE converges at the same rate as if the errors are Gaussian. On the negative side, if p < 1 + 2/a, there are (many) hard models at any entropy level a for which the L2 loss of the LSE converges at a strictly slower rate than other robust estimators. The validity of the above rate relies crucially on the independence of the covariates and the errors. In fact, the L2 loss of the LSE can converge arbitrarily slowly when the independence fails. The key technical ingredient is a new multiplier inequality that gives sharp bounds for the "multiplier empirical process" associated with the LSE. We further give an application to the sparse linear regression model with heavytailed covariates and errors to demonstrate the scope of this new inequality.

AB - We study the performance of the least squares estimator (LSE) in a general nonparametric regression model, when the errors are independent of the covariates but may only have a pth moment (p = 1). In such a heavy-tailed regression setting, we show that if the model satisfies a standard "entropy condition" with exponent a α ϵ (0,2), then the L2 loss of the LSE converges OP n- 1 2+α n-12 + 1 2p. at a rate Such a rate cannot be improved under the entropy condition alone. This rate quantifies both some positive and negative aspects of the LSE in a heavy-tailed regression setting. On the positive side, as long as the errors have p = 1 + 2/a moments, the L2 loss of the LSE converges at the same rate as if the errors are Gaussian. On the negative side, if p < 1 + 2/a, there are (many) hard models at any entropy level a for which the L2 loss of the LSE converges at a strictly slower rate than other robust estimators. The validity of the above rate relies crucially on the independence of the covariates and the errors. In fact, the L2 loss of the LSE can converge arbitrarily slowly when the independence fails. The key technical ingredient is a new multiplier inequality that gives sharp bounds for the "multiplier empirical process" associated with the LSE. We further give an application to the sparse linear regression model with heavytailed covariates and errors to demonstrate the scope of this new inequality.

KW - Heavy-tailed errors

KW - Least squares estimation

KW - Multiplier empirical process

KW - Multiplier inequality

KW - Nonparametric regression

KW - Sparse linear regression

UR - http://www.scopus.com/inward/record.url?scp=85068768612&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85068768612&partnerID=8YFLogxK

U2 - 10.1214/18-AOS1748

DO - 10.1214/18-AOS1748

M3 - Article

AN - SCOPUS:85068768612

SN - 0090-5364

VL - 47

SP - 2286

EP - 2319

JO - Annals of Statistics

JF - Annals of Statistics

IS - 4

ER -