ContinuousTime MetaLearning with Forward Mode Differentiation
Abstract
Drawing inspiration from gradientbased metalearning methods with infinitely small gradient steps, we introduce ContinuousTime MetaLearning (COMLN), a metalearning algorithm where adaptation follows the dynamics of a gradient vector field. Specifically, representations of the inputs are metalearned such that a taskspecific linear classifier is obtained as a solution of an ordinary differential equation (ODE). Treating the learning process as an ODE offers the notable advantage that the length of the trajectory is now continuous, as opposed to a fixed and discrete number of gradient steps. As a consequence, we can optimize the amount of adaptation necessary to solve a new task using stochastic gradient descent, in addition to learning the initial conditions as is standard practice in gradientbased metalearning. Importantly, in order to compute the exact metagradients required for the outerloop updates, we devise an efficient algorithm based on forward mode differentiation, whose memory requirements do not scale with the length of the learning trajectory, thus allowing longer adaptation in constant memory. We provide analytical guarantees for the stability of COMLN, we show empirically its efficiency in terms of runtime and memory usage, and we illustrate its effectiveness on a range of fewshot image classification problems.
 Publication:

arXiv eprints
 Pub Date:
 March 2022
 arXiv:
 arXiv:2203.01443
 Bibcode:
 2022arXiv220301443D
 Keywords:

 Computer Science  Machine Learning