A sparse quantized hopfield network for online-continual memory
Abstract
An important difference between brains and deep neural networks is the way they learn. Nervous systems learn online where a stream of noisy data points are presented in a non-independent, identically distributed way. Further, synaptic plasticity in the brain depends only on information local to synapses. Deep networks, on the other hand, typically use non-local learning algorithms and are trained in an offline, non-noisy, independent, identically distributed setting. Understanding how neural networks learn under the same constraints as the brain is an open problem for neuroscience and neuromorphic computing. A standard approach to this problem has yet to be established. In this paper, we propose that discrete graphical models that learn via an online maximum a posteriori learning algorithm could provide such an approach. We implement this kind of model in a neural network called the Sparse Quantized Hopfield Network. We show our model outperforms state-of-the-art neural networks on associative memory tasks, outperforms these networks in online, continual settings, learns efficiently with noisy inputs, and is better than baselines on an episodic memory task.
- Publication:
-
Nature Communications
- Pub Date:
- May 2024
- DOI:
- 10.1038/s41467-024-46976-4
- arXiv:
- arXiv:2307.15040
- Bibcode:
- 2024NatCo..15.3722A
- Keywords:
-
- Computer Science - Neural and Evolutionary Computing;
- Quantitative Biology - Neurons and Cognition