WigglyEyes: Inferring Eye Movements from Keypress Data

We present a model for inferring where users look during interaction based on keypress data only. Given a key log, it outputs a scanpath that tells, moment-by-moment, how the user had moved eyes while entering those keys. The model can be used as a proxy for human data in cases where collecting real eye tracking data is expensive or impossible. Our technical insight is three-fold: first, we present an inference architecture that considers the individual characteristics of the user, inferred as a low-dimensional parameter vector; second, we present a novel loss function for synchronizing inferred eye movements with the keypresses; third, we train the model using a hybrid approach with both human data and synthetically generated data. The approach can be applied in interactive systems where predictive models of user behavior are available. We report results from evaluation in the challenging case of touchscreen typing, where the model accurately inferred real eye movements.