Concise Explanations of Neural Networks using Adversarial Training
Abstract
We show new connections between adversarial learning and explainability for deep neural networks (DNNs). One form of explanation of the output of a neural network model in terms of its input features, is a vector of featureattributions. Two desirable characteristics of an attributionbased explanation are: (1) $\textit{sparseness}$: the attributions of irrelevant or weakly relevant features should be negligible, thus resulting in $\textit{concise}$ explanations in terms of the significant features, and (2) $\textit{stability}$: it should not vary significantly within a small local neighborhood of the input. Our first contribution is a theoretical exploration of how these two properties (when using attributions based on Integrated Gradients, or IG) are related to adversarial training, for a class of 1layer networks (which includes logistic regression models for binary and multiclass classification); for these networks we show that (a) adversarial training using an $\ell_\infty$bounded adversary produces models with sparse attribution vectors, and (b) natural modeltraining while encouraging stable explanations (via an extra term in the loss function), is equivalent to adversarial training. Our second contribution is an empirical verification of phenomenon (a), which we show, somewhat surprisingly, occurs $\textit{not only}$ $\textit{in 1layer networks}$, $\textit{but also DNNs}$ $\textit{trained on }$ $\textit{standard image datasets}$, and extends beyond IGbased attributions, to those based on DeepSHAP: adversarial training with $\ell_\infty$bounded perturbations yields significantly sparser attribution vectors, with little degradation in performance on natural test data, compared to natural training. Moreover, the sparseness of the attribution vectors is significantly better than that achievable via $\ell_1$regularized natural training.
 Publication:

arXiv eprints
 Pub Date:
 October 2018
 arXiv:
 arXiv:1810.06583
 Bibcode:
 2018arXiv181006583C
 Keywords:

 Computer Science  Machine Learning;
 Statistics  Machine Learning
 EPrint:
 30 pages, 9 figures, 1 table