Local Identifiability of Deep ReLU Neural Networks: the Theory
Abstract
Is a sample rich enough to determine, at least locally, the parameters of a neural network? To answer this question, we introduce a new local parameterization of a given deep ReLU neural network by fixing the values of some of its weights. This allows us to define local lifting operators whose inverses are charts of a smooth manifold of a high dimensional space. The function implemented by the deep ReLU neural network composes the local lifting with a linear operator which depends on the sample. We derive from this convenient representation a geometrical necessary and sufficient condition of local identifiability. Looking at tangent spaces, the geometrical condition provides: 1/ a sharp and testable necessary condition of identifiability and 2/ a sharp and testable sufficient condition of local identifiability. The validity of the conditions can be tested numerically using backpropagation and matrix rank computations.
- Publication:
-
arXiv e-prints
- Pub Date:
- June 2022
- DOI:
- 10.48550/arXiv.2206.07424
- arXiv:
- arXiv:2206.07424
- Bibcode:
- 2022arXiv220607424B
- Keywords:
-
- Mathematics - Statistics Theory;
- Statistics - Machine Learning
- E-Print:
- Advances in Neural Information Processing Systems, Nov 2022, New Orleans, United States