On Algebraic Constructions of Neural Networks with Small Weights
Abstract
Neural gates compute functions based on weighted sums of the input variables. The expressive power of neural gates (number of distinct functions it can compute) depends on the weight sizes and, in general, large weights (exponential in the number of inputs) are required. Studying the tradeoffs among the weight sizes, circuit size and depth is a wellstudied topic both in circuit complexity theory and the practice of neural computation. We propose a new approach for studying these complexity tradeoffs by considering a related algebraic framework. Specifically, given a single linear equation with arbitrary coefficients, we would like to express it using a system of linear equations with smaller (even constant) coefficients. The techniques we developed are based on Siegel's Lemma for the bounds, anticoncentration inequalities for the existential results and extensions of Sylvestertype Hadamard matrices for the constructions. We explicitly construct a constant weight, optimal size matrix to compute the EQUALITY function (checking if two integers expressed in binary are equal). Computing EQUALITY with a single linear equation requires exponentially large weights. In addition, we prove the existence of the bestknown weight size (linear) matrices to compute the COMPARISON function (comparing between two integers expressed in binary). In the context of the circuit complexity theory, our results improve the upper bounds on the weight sizes for the bestknown circuit sizes for EQUALITY and COMPARISON.
 Publication:

arXiv eprints
 Pub Date:
 May 2022
 DOI:
 10.48550/arXiv.2205.08032
 arXiv:
 arXiv:2205.08032
 Bibcode:
 2022arXiv220508032K
 Keywords:

 Computer Science  Computational Complexity;
 Computer Science  Discrete Mathematics;
 Computer Science  Information Theory;
 Computer Science  Machine Learning;
 Computer Science  Neural and Evolutionary Computing