Exploring the Properties and Evolution of Neural Network Eigenspaces during Training
Abstract
In this work we explore the information processing inside neural networks using logistic regression probes \cite{probes} and the saturation metric \cite{featurespace_saturation}. We show that problem difficulty and neural network capacity affect the predictive performance in an antagonistic manner, opening the possibility of detecting over- and under-parameterization of neural networks for a given task. We further show that the observed effects are independent from previously reported pathological patterns like the ``tail pattern'' described in \cite{featurespace_saturation}. Finally we are able to show that saturation patterns converge early during training, allowing for a quicker cycle time during analysis
- Publication:
-
arXiv e-prints
- Pub Date:
- June 2021
- DOI:
- 10.48550/arXiv.2106.09526
- arXiv:
- arXiv:2106.09526
- Bibcode:
- 2021arXiv210609526R
- Keywords:
-
- Computer Science - Machine Learning;
- Computer Science - Artificial Intelligence