Hybrid lemmatization in HuSpaCy
Abstract
Lemmatization is still not a trivial task for morphologically rich languages. Previous studies showed that hybrid architectures usually work better for these languages and can yield great results. This paper presents a hybrid lemmatizer utilizing both a neural model, dictionaries and hand-crafted rules. We introduce a hybrid architecture along with empirical results on a widely used Hungarian dataset. The presented methods are published as three HuSpaCy models.
- Publication:
-
arXiv e-prints
- Pub Date:
- June 2023
- DOI:
- arXiv:
- arXiv:2306.07636
- Bibcode:
- 2023arXiv230607636B
- Keywords:
-
- Computer Science - Computation and Language;
- 68T50;
- I.2.7
- E-Print:
- published at the conference XIX. Magyar Sz\'am\'it\'og\'epes Nyelv\'eszeti Konferencia (XIX. Hungarian Computational Linguistics Conference)