Variable selection and estimation with the seamless-L0 penalty

Lee Dicker, Baosheng Huang, Xihong Lin

Research output: Contribution to journalArticle

34 Citations (Scopus)

Abstract

Penalized least squares procedures that directly penalize the number of variables in a regression model (L0 penalized least squares procedures) enjoy nice theoretical properties and are intuitively appealing. On the other hand, L0 penalized least squares methods also have significant drawbacks in that implementation is NP-hard and not computationally feasible when the number of variables is even moderately large. One of the challenges is the discontinuity of the L0 penalty. We propose the seamless-L0 (SELO) penalty, a smooth function on [0;∞) that very closely resembles the L0 penalty. The SELO penalized least squares procedure is shown to consistently select the correct model and is asymptotically normal, provided the number of variables grows more slowly than the number of observations. SELO is efficiently implemented using a coordinate descent algorithm. Since tuning parameter selection is crucial to the performance of the SELO procedure, we propose a BIC-like tuning parameter selection method for SELO, and show that it consistently identifies the correct model while allowing the number of variables to diverge. Simulation results show that the SELO procedure with BIC tuning parameter selection performs well in a variety of settings - outperforming other popular penalized least squares procedures by a substantial margin. Using SELO, we analyze a publicly available HIV drug resistance and mutation dataset and obtain interpretable results.

Original languageEnglish (US)
Pages (from-to)929-962
Number of pages34
JournalStatistica Sinica
Volume23
Issue number2
DOIs
StatePublished - Apr 1 2013

Fingerprint

Penalized Least Squares
Variable Selection
Penalty
Parameter Selection
Tuning
Coordinate Descent
Drug Resistance
Descent Algorithm
Diverge
Least Square Method
Smooth function
Margin
Variable selection
Discontinuity
Regression Model
Mutation
NP-complete problem
Least squares
Model
Simulation

All Science Journal Classification (ASJC) codes

  • Statistics and Probability
  • Statistics, Probability and Uncertainty

Keywords

  • BIC
  • Coordinate descent
  • Oracle property
  • Penalized least squares
  • Tuning parameter selection

Cite this

Dicker, Lee ; Huang, Baosheng ; Lin, Xihong. / Variable selection and estimation with the seamless-L0 penalty. In: Statistica Sinica. 2013 ; Vol. 23, No. 2. pp. 929-962.
@article{a2910031fdf5425d8ccaf4d40d634bfe,
title = "Variable selection and estimation with the seamless-L0 penalty",
abstract = "Penalized least squares procedures that directly penalize the number of variables in a regression model (L0 penalized least squares procedures) enjoy nice theoretical properties and are intuitively appealing. On the other hand, L0 penalized least squares methods also have significant drawbacks in that implementation is NP-hard and not computationally feasible when the number of variables is even moderately large. One of the challenges is the discontinuity of the L0 penalty. We propose the seamless-L0 (SELO) penalty, a smooth function on [0;∞) that very closely resembles the L0 penalty. The SELO penalized least squares procedure is shown to consistently select the correct model and is asymptotically normal, provided the number of variables grows more slowly than the number of observations. SELO is efficiently implemented using a coordinate descent algorithm. Since tuning parameter selection is crucial to the performance of the SELO procedure, we propose a BIC-like tuning parameter selection method for SELO, and show that it consistently identifies the correct model while allowing the number of variables to diverge. Simulation results show that the SELO procedure with BIC tuning parameter selection performs well in a variety of settings - outperforming other popular penalized least squares procedures by a substantial margin. Using SELO, we analyze a publicly available HIV drug resistance and mutation dataset and obtain interpretable results.",
keywords = "BIC, Coordinate descent, Oracle property, Penalized least squares, Tuning parameter selection",
author = "Lee Dicker and Baosheng Huang and Xihong Lin",
year = "2013",
month = "4",
day = "1",
doi = "10.5705/ss.2011.074",
language = "English (US)",
volume = "23",
pages = "929--962",
journal = "Statistica Sinica",
issn = "1017-0405",
publisher = "Institute of Statistical Science",
number = "2",

}

Variable selection and estimation with the seamless-L0 penalty. / Dicker, Lee; Huang, Baosheng; Lin, Xihong.

In: Statistica Sinica, Vol. 23, No. 2, 01.04.2013, p. 929-962.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Variable selection and estimation with the seamless-L0 penalty

AU - Dicker, Lee

AU - Huang, Baosheng

AU - Lin, Xihong

PY - 2013/4/1

Y1 - 2013/4/1

N2 - Penalized least squares procedures that directly penalize the number of variables in a regression model (L0 penalized least squares procedures) enjoy nice theoretical properties and are intuitively appealing. On the other hand, L0 penalized least squares methods also have significant drawbacks in that implementation is NP-hard and not computationally feasible when the number of variables is even moderately large. One of the challenges is the discontinuity of the L0 penalty. We propose the seamless-L0 (SELO) penalty, a smooth function on [0;∞) that very closely resembles the L0 penalty. The SELO penalized least squares procedure is shown to consistently select the correct model and is asymptotically normal, provided the number of variables grows more slowly than the number of observations. SELO is efficiently implemented using a coordinate descent algorithm. Since tuning parameter selection is crucial to the performance of the SELO procedure, we propose a BIC-like tuning parameter selection method for SELO, and show that it consistently identifies the correct model while allowing the number of variables to diverge. Simulation results show that the SELO procedure with BIC tuning parameter selection performs well in a variety of settings - outperforming other popular penalized least squares procedures by a substantial margin. Using SELO, we analyze a publicly available HIV drug resistance and mutation dataset and obtain interpretable results.

AB - Penalized least squares procedures that directly penalize the number of variables in a regression model (L0 penalized least squares procedures) enjoy nice theoretical properties and are intuitively appealing. On the other hand, L0 penalized least squares methods also have significant drawbacks in that implementation is NP-hard and not computationally feasible when the number of variables is even moderately large. One of the challenges is the discontinuity of the L0 penalty. We propose the seamless-L0 (SELO) penalty, a smooth function on [0;∞) that very closely resembles the L0 penalty. The SELO penalized least squares procedure is shown to consistently select the correct model and is asymptotically normal, provided the number of variables grows more slowly than the number of observations. SELO is efficiently implemented using a coordinate descent algorithm. Since tuning parameter selection is crucial to the performance of the SELO procedure, we propose a BIC-like tuning parameter selection method for SELO, and show that it consistently identifies the correct model while allowing the number of variables to diverge. Simulation results show that the SELO procedure with BIC tuning parameter selection performs well in a variety of settings - outperforming other popular penalized least squares procedures by a substantial margin. Using SELO, we analyze a publicly available HIV drug resistance and mutation dataset and obtain interpretable results.

KW - BIC

KW - Coordinate descent

KW - Oracle property

KW - Penalized least squares

KW - Tuning parameter selection

UR - http://www.scopus.com/inward/record.url?scp=84884275076&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84884275076&partnerID=8YFLogxK

U2 - 10.5705/ss.2011.074

DO - 10.5705/ss.2011.074

M3 - Article

AN - SCOPUS:84884275076

VL - 23

SP - 929

EP - 962

JO - Statistica Sinica

JF - Statistica Sinica

SN - 1017-0405

IS - 2

ER -