Part-of-Speech Tagging with Minimal Lexicalization
Abstract
We use a Dynamic Bayesian Network to represent compactly a variety of sublexical and contextual features relevant to Part-of-Speech (PoS) tagging. The outcome is a flexible tagger (LegoTag) with state-of-the-art performance (3.6% error on a benchmark corpus). We explore the effect of eliminating redundancy and radically reducing the size of feature vocabularies. We find that a small but linguistically motivated set of suffixes results in improved cross-corpora generalization. We also show that a minimal lexicon limited to function words is sufficient to ensure reasonable performance.
- Publication:
-
arXiv e-prints
- Pub Date:
- December 2003
- DOI:
- 10.48550/arXiv.cs/0312060
- arXiv:
- arXiv:cs/0312060
- Bibcode:
- 2003cs.......12060S
- Keywords:
-
- Computer Science - Computation and Language;
- Computer Science - Machine Learning;
- I.2.7
- E-Print:
- 10 pages text