Thank you BART! Rewarding Pre-Trained Models Improves Formality Style Transfer
Abstract
Scarcity of parallel data causes formality style transfer models to have scarce success in preserving content. We show that fine-tuning pre-trained language (GPT-2) and sequence-to-sequence (BART) models boosts content preservation, and that this is possible even with limited amounts of parallel data. Augmenting these models with rewards that target style and content -- the two core aspects of the task -- we achieve a new state-of-the-art.
- Publication:
-
arXiv e-prints
- Pub Date:
- May 2021
- DOI:
- 10.48550/arXiv.2105.06947
- arXiv:
- arXiv:2105.06947
- Bibcode:
- 2021arXiv210506947L
- Keywords:
-
- Computer Science - Computation and Language