Self-supervised Fine-tuning for Improved Content Representations by Speaker-invariant Clustering
Abstract
Self-supervised speech representation models have succeeded in various tasks, but improving them for content-related problems using unlabeled data is challenging. We propose speaker-invariant clustering (Spin), a novel self-supervised learning method that clusters speech representations and performs swapped prediction between the original and speaker-perturbed utterances. Spin disentangles speaker information and preserves content representations with just 45 minutes of fine-tuning on a single GPU. Spin improves pre-trained networks and outperforms prior methods in speech recognition and acoustic unit discovery.
- Publication:
-
arXiv e-prints
- Pub Date:
- May 2023
- DOI:
- 10.48550/arXiv.2305.11072
- arXiv:
- arXiv:2305.11072
- Bibcode:
- 2023arXiv230511072C
- Keywords:
-
- Computer Science - Computation and Language;
- Electrical Engineering and Systems Science - Audio and Speech Processing
- E-Print:
- Accepted to Interspeech 2023