Nonequilibrium thermodynamics of self-supervised learning
Abstract
Self-supervised learning (SSL) of energy based models has an intuitive relation to equilibrium thermodynamics because the softmax layer, mapping energies to probabilities, is a Gibbs distribution. However, in what way SSL is a thermodynamic process? We show that some SSL paradigms behave as a thermodynamic composite system formed by representations and self-labels in contact with a nonequilibrium reservoir. Moreover, this system is subjected to usual thermodynamic cycles, such as adiabatic expansion and isochoric heating, resulting in a generalized Gibbs ensemble (GGE). In this picture, we show that learning is seen as a demon that operates in cycles using feedback measurements to extract negative work from the system. As applications, we examine some SSL algorithms using this idea.
- Publication:
-
Physics Letters A
- Pub Date:
- December 2021
- DOI:
- 10.1016/j.physleta.2021.127756
- arXiv:
- arXiv:2106.08981
- Bibcode:
- 2021PhLA..41927756S
- Keywords:
-
- Statistical mechanics;
- Artificial intelligence;
- Machine learning;
- Condensed Matter - Statistical Mechanics;
- Computer Science - Artificial Intelligence;
- Computer Science - Machine Learning
- E-Print:
- 6 pages, 1 figure