When we read, our eyes do not glide smoothly across the page like a scanner. Instead, they move in rapid jumps called saccades, punctuated by brief pauses known as fixations. Most interestingly, our eyes frequently jump backward to words we have already passed. These backward movements, or regressions, were once viewed primarily as signs of simple reading failure. However, a growing body of research suggests that regressions are a sophisticated cognitive mechanism used to resolve linguistic ambiguity, verify syntax, and detect subtle semantic cues like irony.
By treating these eye movements as a "digital exhaust" of human cognition, researchers are now using gaze data to improve how machines understand language.
The Syntax of a Glance
The core intuition behind this research is that human eye movements are highly sensitive to the underlying structure of a sentence. As Barrett et al. argue, reading patterns are not random; they are shaped by the syntactic complexity of the text.
This relationship is so strong that gaze features can actually predict a word's part-of-speech. Barrett et al. demonstrated that the way a reader fixates on a word provides a measurable trace of its syntactic category, such as whether it is a noun or a verb. This implicit signal is particularly valuable when training data is scarce, allowing models to learn linguistic categories more effectively by observing how humans process them.
Regressions as Structural Probes
If fixations tell us about individual words, regressions tell us about the relationships between words. When a reader encounters a word that doesn't immediately fit the expected structure, their eyes dart back to find the "head" of the phrase or a related dependent.
Lopopolo et al. found that regressions are significantly more likely to land on words that are syntactically connected to the current word in a dependency tree. This suggests that regressions aren't just random backtracking; they are a targeted search for the information needed to build a mental map of the sentence. By incorporating these signals, Agrawal et al. showed that syntactic parsers can become much more robust, especially in "delexicalized" settings where the model cannot rely on the words themselves and must instead rely on structural cues.
Resolving Semantic Conflict
Beyond grammar, regressions serve as a primary tool for resolving semantic incongruity. This is most evident in the detection of sarcasm—the "dark matter" of language. Sarcasm often involves a conflict between the literal meaning of words and the intended context.
Mishra et al. observed that when readers encounter sarcasm, they frequently exhibit long regressions, returning to earlier parts of the sentence to reconcile the conflicting information. These gaze patterns are so distinct that they can be used to predict not only if a sentence is sarcastic, but also whether the reader successfully "got" the joke. As Kanojia et al. summarize in their survey, these backward eye movements act as a proxy for the cognitive re-evaluation required to interpret non-literal language.
Evaluating Text Quality
Because regressions signal processing difficulty, they also serve as a natural metric for text quality and grammaticality.
- Grammaticality: Klerke et al. found that the proportion of regressions is a strong indicator of ungrammaticality in automatically generated text. If a machine-compressed sentence is broken, the reader’s eyes will reflexively stutter and look back.
- Cohesion: Mathias et al. demonstrated that regression features are essential for assessing text cohesion. To understand how sentences link together, readers must look back to trace connections, making gaze a more reliable indicator of quality than text-based features alone.
By integrating these physiological signals into machine learning models, as discussed in the LANTERN workshop, researchers are moving toward AI that doesn't just process strings of symbols, but mimics the holistic, non-linear way that humans construct meaning.
Key Foundational Work
Looking hard: Eye tracking for detecting grammaticality of automatically compressed sentences (2015) — contributed 2 concept elements to this direction.
Harnessing Cognitive Features for Sarcasm Detection (2016) — contributed 2 concept elements to this direction.
Reading behavior predicts syntactic categories (2015) — contributed 1 concept elements to this direction.