Abstract
One major bottleneck in conversational systems is their incapability in interpreting unexpected user language inputs such as out-of-vocabulary (OOV) words. To overcome this problem, conversational systems must be able to learn new words automatically during human-machine conversation. Motivated by psycholinguistic findings on eye gaze and human language processing, we are developing techniques to incorporate human eye gaze for automatic word acquisition in multimodal conversational systems. This paper investigates the use of temporal alignment between speech and eye gaze and the use of domain knowledge in word acquisition. Our experiment results indicate that eye gaze provides a potential channel for automatically acquiring new words. The use of extra temporal and domain knowledge can significantly improve acquisition performance.
1. Introduction
The "Out-of-Vocabulary" (OOV) problem remains a significant hurdle for conversational AI. When a user employs a term outside the system's predefined lexicon—such as calling a "dresser" a "bureau"—the interaction typically fails. Manually cataloging every possible synonym is impractical. This research explores automatic word acquisition: the ability of a system to learn that a novel acoustic signal refers to a specific 3D object by observing natural human behavior, specifically eye gaze.
The core challenge lies in "grounding"—associating a continuous, noisy stream of eye-tracking data with a sequence of spoken words to correctly identify which word refers to which object.
2. Methodology: Word Acquisition as Translation
The proposed approach treats word acquisition as a statistical machine translation problem. Rather than translating between natural languages, the system "translates" a sequence of fixated visual entities into a sequence of spoken words.
2.1 Statistical Alignment
Building upon IBM Translation Models, the researchers introduced two novel constraints to refine the alignment:
- Temporal Alignment (Model-2t): Psycholinguistic studies suggest humans fixate on an object approximately 600-1000ms before naming it. The system utilizes a probability distribution that favors aligning words with gaze fixations occurring shortly before the word's onset.
- Semantic Relatedness (Model-2s): Objects in the 3D environment are linked to WordNet synsets. This allows the system to use semantic knowledge as a "sanity check," favoring candidate words that are semantically related to the object's category (e.g., associating "bureau" with a furniture object rather than a light fixture).
3. Temporal and Semantic Insights
A key finding of this work is the "Look-Before-Leap" principle. Empirical data confirms that eye gaze rarely follows the spoken word; instead, the eyes have usually moved on to the next point of interest by the time a word is uttered.
As shown in the histogram above, the vast majority of successful alignments occur when the gaze fixation precedes the word onset (negative temporal distance). Modeling this absolute timestamp is significantly more effective than relying on the relative position of a word within a sentence.
4. Experimental Results
The system was evaluated using a custom dataset involving a 3D room decoration simulator and eye-tracking data from seven users. Performance was measured using the Mean Reciprocal Rank Rate (MRRR):
- Baseline (Model-1): Achieved an MRRR of 0.59.
- Temporal Model (Model-2t): Improved performance to 0.69, highlighting the strength of the timing signal.
- Semantic Rescoring (Model-2t-r): The top-performing configuration reached an MRRR of 0.75.
These results demonstrate that incorporating temporal and domain knowledge significantly enhances the system's ability to acquire new vocabulary from multimodal inputs.
5. Discussion and Future Work
While promising, several challenges remain. Data scarcity for infrequent words (the "long tail") makes learning difficult. Furthermore, the current model relies on a predefined domain linked to WordNet; scaling to open-world environments would require more sophisticated visual-semantic mapping. Future research will explore real-time active learning, where the system can seek clarification when it encounters high uncertainty during word acquisition.
Summary updated