A Formal Analysis of Multimodal Referring Strategies Under Common Ground
Abstract
In this paper, we present an analysis of computationally generated mixed-modality definite referring expressions using combinations of gesture and linguistic descriptions. In doing so, we expose some striking formal semantic properties of the interactions between gesture and language, conditioned on the introduction of content into the common ground between the (computational) speaker and (human) viewer, and demonstrate how these formal features can contribute to training better models to predict viewer judgment of referring expressions, and potentially to the generation of more natural and informative referring expressions.
- Publication:
-
arXiv e-prints
- Pub Date:
- March 2020
- DOI:
- 10.48550/arXiv.2003.07385
- arXiv:
- arXiv:2003.07385
- Bibcode:
- 2020arXiv200307385K
- Keywords:
-
- Computer Science - Computation and Language;
- Computer Science - Artificial Intelligence
- E-Print:
- 9 pages (incl refs), 7 figures, 3 tables, proceedings of LREC 2020 (postponed due to COVID-19)