TY - GEN
T1 - Correlations between deep neural network model coverage criteria and model quality
AU - Yan, Shenao
AU - Tao, Guanhong
AU - Liu, Xuwei
AU - Zhai, Juan
AU - Ma, Shiqing
AU - Xu, Lei
AU - Zhang, Xiangyu
N1 - Publisher Copyright:
© 2020 ACM.
PY - 2020/11/8
Y1 - 2020/11/8
N2 - Inspired by the great success of using code coverage as guidance in software testing, a lot of neural network coverage criteria have been proposed to guide testing of neural network models (e.g., model accuracy under adversarial attacks). However, while the monotonic relation between code coverage and software quality has been supported by many seminal studies in software engineering, it remains largely unclear whether similar monotonicity exists between neural network model coverage and model quality. This paper sets out to answer this question. Specifically, this paper studies the correlation between DNN model quality and coverage criteria, effects of coverage guided adversarial example generation compared with gradient decent based methods, effectiveness of coverage based retraining compared with existing adversarial training, and the internal relationships among coverage criteria.
AB - Inspired by the great success of using code coverage as guidance in software testing, a lot of neural network coverage criteria have been proposed to guide testing of neural network models (e.g., model accuracy under adversarial attacks). However, while the monotonic relation between code coverage and software quality has been supported by many seminal studies in software engineering, it remains largely unclear whether similar monotonicity exists between neural network model coverage and model quality. This paper sets out to answer this question. Specifically, this paper studies the correlation between DNN model quality and coverage criteria, effects of coverage guided adversarial example generation compared with gradient decent based methods, effectiveness of coverage based retraining compared with existing adversarial training, and the internal relationships among coverage criteria.
KW - Deep Neural Networks
KW - Software Testing
UR - http://www.scopus.com/inward/record.url?scp=85097200783&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85097200783&partnerID=8YFLogxK
U2 - 10.1145/3368089.3409671
DO - 10.1145/3368089.3409671
M3 - Conference contribution
AN - SCOPUS:85097200783
T3 - ESEC/FSE 2020 - Proceedings of the 28th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering
SP - 775
EP - 787
BT - ESEC/FSE 2020 - Proceedings of the 28th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering
A2 - Devanbu, Prem
A2 - Cohen, Myra
A2 - Zimmermann, Thomas
PB - Association for Computing Machinery, Inc
T2 - 28th ACM Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2020
Y2 - 8 November 2020 through 13 November 2020
ER -