Invariance and identifiability issues for word embeddings
Abstract
Word embeddings are commonly obtained as optimizers of a criterion function $f$ of a text corpus, but assessed on wordtask performance using a different evaluation function $g$ of the test data. We contend that a possible source of disparity in performance on tasks is the incompatibility between classes of transformations that leave $f$ and $g$ invariant. In particular, word embeddings defined by $f$ are not unique; they are defined only up to a class of transformations to which $f$ is invariant, and this class is larger than the class to which $g$ is invariant. One implication of this is that the apparent superiority of one word embedding over another, as measured by word task performance, may largely be a consequence of the arbitrary elements selected from the respective solution sets. We provide a formal treatment of the above identifiability issue, present some numerical examples, and discuss possible resolutions.
 Publication:

arXiv eprints
 Pub Date:
 November 2019
 arXiv:
 arXiv:1911.02656
 Bibcode:
 2019arXiv191102656C
 Keywords:

 Statistics  Machine Learning;
 Computer Science  Computation and Language;
 Computer Science  Machine Learning;
 Statistics  Computation
 EPrint:
 NIPS 2019