Closedform Continuoustime Neural Models
Abstract
Continuoustime neural processes are performant sequential decisionmakers that are built by differential equations (DE). However, their expressive power when they are deployed on computers is bottlenecked by numerical DE solvers. This limitation has significantly slowed down the scaling and understanding of numerous natural physical phenomena such as the dynamics of nervous systems. Ideally, we would circumvent this bottleneck by solving the given dynamical system in closed form. This is known to be intractable in general. Here, we show it is possible to closely approximate the interaction between neurons and synapses  the building blocks of natural and artificial neural networks  constructed by liquid timeconstant networks (LTCs) efficiently in closedform. To this end, we compute a tightlybounded approximation of the solution of an integral appearing in LTCs' dynamics, that has had no known closedform solution so far. This closedform solution substantially impacts the design of continuoustime and continuousdepth neural models; for instance, since time appears explicitly in closedform, the formulation relaxes the need for complex numerical solvers. Consequently, we obtain models that are between one and five orders of magnitude faster in training and inference compared to differential equationbased counterparts. More importantly, in contrast to ODEbased continuous networks, closedform networks can scale remarkably well compared to other deep learning instances. Lastly, as these models are derived from liquid networks, they show remarkable performance in time series modeling, compared to advanced recurrent models.
 Publication:

arXiv eprints
 Pub Date:
 June 2021
 DOI:
 10.48550/arXiv.2106.13898
 arXiv:
 arXiv:2106.13898
 Bibcode:
 2021arXiv210613898H
 Keywords:

 Computer Science  Machine Learning;
 Computer Science  Artificial Intelligence;
 Computer Science  Neural and Evolutionary Computing;
 Computer Science  Robotics;
 Mathematics  Dynamical Systems
 EPrint:
 40 pages