In many contexts, creating mappings for gestural interactions can form part of an artistic process. Creators seeking a mapping that is expressive, novel, and affords them a sense of authorship may not know how to program it up in a signal processing patch. Tools like Wekinator and MIMIC allow creators to use supervised machine learning to learn mappings from example input/output pairings. However, a creator may know a good mapping when they encounter it yet start with little sense of what the inputs or outputs should be. We call this an open-ended mapping process. Addressing this need, we introduce the latent mapping, which leverages the latent space of an unsupervised machine learning algorithm such as a Variational Autoencoder trained on a corpus of unlabelled gestural data from the creator. We illustrate it with Sonified Body, a system mapping full-body movement to sound which we explore in a residency with three dancers.
- Pub Date:
- June 2021
- Computer Science - Human-Computer Interaction;
- Computer Science - Multimedia
- Published at the International Conference on New Interfaces for Musical Expression, June 2021. 3000 word short paper. 5 figures plus video which may be seen at https://timmb.com/sonified-body-r-and-d-lab