Brain Decodes Deep Nets
Abstract
We developed a tool for visualizing and analyzing large pre-trained vision models by mapping them onto the brain, thus exposing their hidden inside. Our innovation arises from a surprising usage of brain encoding: predicting brain fMRI measurements in response to images. We report two findings. First, explicit mapping between the brain and deep-network features across dimensions of space, layers, scales, and channels is crucial. This mapping method, FactorTopy, is plug-and-play for any deep-network; with it, one can paint a picture of the network onto the brain (literally!). Second, our visualization shows how different training methods matter: they lead to remarkable differences in hierarchical organization and scaling behavior, growing with more data or network capacity. It also provides insight into fine-tuning: how pre-trained models change when adapting to small datasets. We found brain-like hierarchically organized network suffer less from catastrophic forgetting after fine-tuned.
- Publication:
-
arXiv e-prints
- Pub Date:
- December 2023
- DOI:
- 10.48550/arXiv.2312.01280
- arXiv:
- arXiv:2312.01280
- Bibcode:
- 2023arXiv231201280Y
- Keywords:
-
- Computer Science - Computer Vision and Pattern Recognition
- E-Print:
- Website: see https://huzeyann.github.io/brain-decodes-deep-nets . Code: see https://github.com/huzeyann/BrainDecodesDeepNets