TY - GEN
T1 - Correlations between deep neural network model coverage criteria and model quality
AU - Yan, Shenao
AU - Tao, Guanhong
AU - Liu, Xuwei
AU - Zhai, Juan
AU - Ma, Shiqing
AU - Xu, Lei
AU - Zhang, Xiangyu
N1 - Funding Information:
We sincerely thank the anonymous reviewers for their constructive comments, feedbacks and suggestions. We also thank the authors of tools used in this paper for their open source efforts, without which this research is impossible. This research was supported, in part by NSF China No.61802166, DARPA FA8650-15-C-7562, NSF 1748764, NSF 1901242, NSF 1910300, ONR N000141410468, ONR N000141712947, and Sandia National Lab under award 1701331. Any opinions, findings, and conclusions made in this paper are those of the authors only and do not necessarily reflect the views of our sponsors.
Publisher Copyright:
© 2020 ACM.
PY - 2020/11/8
Y1 - 2020/11/8
N2 - Inspired by the great success of using code coverage as guidance in software testing, a lot of neural network coverage criteria have been proposed to guide testing of neural network models (e.g., model accuracy under adversarial attacks). However, while the monotonic relation between code coverage and software quality has been supported by many seminal studies in software engineering, it remains largely unclear whether similar monotonicity exists between neural network model coverage and model quality. This paper sets out to answer this question. Specifically, this paper studies the correlation between DNN model quality and coverage criteria, effects of coverage guided adversarial example generation compared with gradient decent based methods, effectiveness of coverage based retraining compared with existing adversarial training, and the internal relationships among coverage criteria.
AB - Inspired by the great success of using code coverage as guidance in software testing, a lot of neural network coverage criteria have been proposed to guide testing of neural network models (e.g., model accuracy under adversarial attacks). However, while the monotonic relation between code coverage and software quality has been supported by many seminal studies in software engineering, it remains largely unclear whether similar monotonicity exists between neural network model coverage and model quality. This paper sets out to answer this question. Specifically, this paper studies the correlation between DNN model quality and coverage criteria, effects of coverage guided adversarial example generation compared with gradient decent based methods, effectiveness of coverage based retraining compared with existing adversarial training, and the internal relationships among coverage criteria.
KW - Deep Neural Networks
KW - Software Testing
UR - http://www.scopus.com/inward/record.url?scp=85097200783&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85097200783&partnerID=8YFLogxK
U2 - 10.1145/3368089.3409671
DO - 10.1145/3368089.3409671
M3 - Conference contribution
AN - SCOPUS:85097200783
T3 - ESEC/FSE 2020 - Proceedings of the 28th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering
SP - 775
EP - 787
BT - ESEC/FSE 2020 - Proceedings of the 28th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering
A2 - Devanbu, Prem
A2 - Cohen, Myra
A2 - Zimmermann, Thomas
PB - Association for Computing Machinery, Inc
T2 - 28th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020
Y2 - 8 November 2020 through 13 November 2020
ER -