Theoretical properties for neural networks with weight matrices of low displacement rank

Liang Zhao, Siyu Liao, Yanzhi Wang, Zhe Li, Jian Tang, Bo Yuan

Research output: Chapter in Book/Report/Conference proceedingConference contribution

2 Citations (Scopus)

Abstract

Recently low displacement rank (LDR) matrices, or so-called structured matrices, have been proposed to compress large-scale neural networks. Empirical results have shown that neural networks with weight matrices of LDR matrices, referred as LDR neural networks, can achieve significant reduction in space and computational complexity while retaining high accuracy. We formally study LDR matrices in deep learning. First, we prove the universal approximation property of LDR neural networks with a mild condition on the displacement operators. We then show that the error bounds of LDR neural networks are as efficient as general neural networks with both single-layer and multiple-layer structure. Finally, we propose back-propagation based training algorithm for general LDR neural networks.

Original languageEnglish (US)
Title of host publication34th International Conference on Machine Learning, ICML 2017
PublisherInternational Machine Learning Society (IMLS)
Pages6186-6194
Number of pages9
ISBN (Electronic)9781510855144
StatePublished - Jan 1 2017
Externally publishedYes
Event34th International Conference on Machine Learning, ICML 2017 - Sydney, Australia
Duration: Aug 6 2017Aug 11 2017

Publication series

Name34th International Conference on Machine Learning, ICML 2017
Volume8

Other

Other34th International Conference on Machine Learning, ICML 2017
CountryAustralia
CitySydney
Period8/6/178/11/17

Fingerprint

Neural networks
Backpropagation
Computational complexity

All Science Journal Classification (ASJC) codes

  • Computational Theory and Mathematics
  • Human-Computer Interaction
  • Software

Cite this

Zhao, L., Liao, S., Wang, Y., Li, Z., Tang, J., & Yuan, B. (2017). Theoretical properties for neural networks with weight matrices of low displacement rank. In 34th International Conference on Machine Learning, ICML 2017 (pp. 6186-6194). (34th International Conference on Machine Learning, ICML 2017; Vol. 8). International Machine Learning Society (IMLS).
Zhao, Liang ; Liao, Siyu ; Wang, Yanzhi ; Li, Zhe ; Tang, Jian ; Yuan, Bo. / Theoretical properties for neural networks with weight matrices of low displacement rank. 34th International Conference on Machine Learning, ICML 2017. International Machine Learning Society (IMLS), 2017. pp. 6186-6194 (34th International Conference on Machine Learning, ICML 2017).
@inproceedings{d7d170e009414356b8d66935a9a78f26,
title = "Theoretical properties for neural networks with weight matrices of low displacement rank",
abstract = "Recently low displacement rank (LDR) matrices, or so-called structured matrices, have been proposed to compress large-scale neural networks. Empirical results have shown that neural networks with weight matrices of LDR matrices, referred as LDR neural networks, can achieve significant reduction in space and computational complexity while retaining high accuracy. We formally study LDR matrices in deep learning. First, we prove the universal approximation property of LDR neural networks with a mild condition on the displacement operators. We then show that the error bounds of LDR neural networks are as efficient as general neural networks with both single-layer and multiple-layer structure. Finally, we propose back-propagation based training algorithm for general LDR neural networks.",
author = "Liang Zhao and Siyu Liao and Yanzhi Wang and Zhe Li and Jian Tang and Bo Yuan",
year = "2017",
month = "1",
day = "1",
language = "English (US)",
series = "34th International Conference on Machine Learning, ICML 2017",
publisher = "International Machine Learning Society (IMLS)",
pages = "6186--6194",
booktitle = "34th International Conference on Machine Learning, ICML 2017",

}

Zhao, L, Liao, S, Wang, Y, Li, Z, Tang, J & Yuan, B 2017, Theoretical properties for neural networks with weight matrices of low displacement rank. in 34th International Conference on Machine Learning, ICML 2017. 34th International Conference on Machine Learning, ICML 2017, vol. 8, International Machine Learning Society (IMLS), pp. 6186-6194, 34th International Conference on Machine Learning, ICML 2017, Sydney, Australia, 8/6/17.

Theoretical properties for neural networks with weight matrices of low displacement rank. / Zhao, Liang; Liao, Siyu; Wang, Yanzhi; Li, Zhe; Tang, Jian; Yuan, Bo.

34th International Conference on Machine Learning, ICML 2017. International Machine Learning Society (IMLS), 2017. p. 6186-6194 (34th International Conference on Machine Learning, ICML 2017; Vol. 8).

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Theoretical properties for neural networks with weight matrices of low displacement rank

AU - Zhao, Liang

AU - Liao, Siyu

AU - Wang, Yanzhi

AU - Li, Zhe

AU - Tang, Jian

AU - Yuan, Bo

PY - 2017/1/1

Y1 - 2017/1/1

N2 - Recently low displacement rank (LDR) matrices, or so-called structured matrices, have been proposed to compress large-scale neural networks. Empirical results have shown that neural networks with weight matrices of LDR matrices, referred as LDR neural networks, can achieve significant reduction in space and computational complexity while retaining high accuracy. We formally study LDR matrices in deep learning. First, we prove the universal approximation property of LDR neural networks with a mild condition on the displacement operators. We then show that the error bounds of LDR neural networks are as efficient as general neural networks with both single-layer and multiple-layer structure. Finally, we propose back-propagation based training algorithm for general LDR neural networks.

AB - Recently low displacement rank (LDR) matrices, or so-called structured matrices, have been proposed to compress large-scale neural networks. Empirical results have shown that neural networks with weight matrices of LDR matrices, referred as LDR neural networks, can achieve significant reduction in space and computational complexity while retaining high accuracy. We formally study LDR matrices in deep learning. First, we prove the universal approximation property of LDR neural networks with a mild condition on the displacement operators. We then show that the error bounds of LDR neural networks are as efficient as general neural networks with both single-layer and multiple-layer structure. Finally, we propose back-propagation based training algorithm for general LDR neural networks.

UR - http://www.scopus.com/inward/record.url?scp=85048550180&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85048550180&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:85048550180

T3 - 34th International Conference on Machine Learning, ICML 2017

SP - 6186

EP - 6194

BT - 34th International Conference on Machine Learning, ICML 2017

PB - International Machine Learning Society (IMLS)

ER -

Zhao L, Liao S, Wang Y, Li Z, Tang J, Yuan B. Theoretical properties for neural networks with weight matrices of low displacement rank. In 34th International Conference on Machine Learning, ICML 2017. International Machine Learning Society (IMLS). 2017. p. 6186-6194. (34th International Conference on Machine Learning, ICML 2017).