GOVERN: Gradient Orientation Vote Ensemble for Multi-Teacher Reinforced Distillation
Abstract
Pre-trained language models have become an integral component of question-answering systems, achieving remarkable performance. For practical deployment, it is critical to carry out knowledge distillation to preserve high performance under computational constraints. In this paper, we address a key question: given the importance of unsupervised distillation for student performance, how does one effectively ensemble knowledge from multiple teachers at this stage without the guidance of ground-truth labels? We propose a novel algorithm, GOVERN, to tackle this issue. GOVERN has demonstrated significant improvements in both offline and online experiments. The proposed algorithm has been successfully deployed in a real-world commercial question-answering system.
- Publication:
-
arXiv e-prints
- Pub Date:
- May 2024
- DOI:
- 10.48550/arXiv.2405.03764
- arXiv:
- arXiv:2405.03764
- Bibcode:
- 2024arXiv240503764Z
- Keywords:
-
- Computer Science - Computation and Language;
- Computer Science - Information Retrieval