Domain-specific or Uncertainty-aware models: Does it really make a difference for biomedical text classification?
Abstract
The success of pretrained language models (PLMs) across a spate of use-cases has led to significant investment from the NLP community towards building domain-specific foundational models. On the other hand, in mission critical settings such as biomedical applications, other aspects also factor in-chief of which is a model's ability to produce reasonable estimates of its own uncertainty. In the present study, we discuss these two desiderata through the lens of how they shape the entropy of a model's output probability distribution. We find that domain specificity and uncertainty awareness can often be successfully combined, but the exact task at hand weighs in much more strongly.
- Publication:
-
arXiv e-prints
- Pub Date:
- July 2024
- DOI:
- 10.48550/arXiv.2407.12626
- arXiv:
- arXiv:2407.12626
- Bibcode:
- 2024arXiv240712626S
- Keywords:
-
- Computer Science - Computation and Language
- E-Print:
- BioNLP 2024