TL;DR
This paper introduces "conditional modality fusion," a novel approach that learns to identify when gestures are truly informative for natural language tasks like coreference resolution. By treating gesture salience as a hidden variable, the model dynamically incorporates gestures only when they enhance understanding, leading to significant performance improvements and enabling new applications like video summarization.
What Problem Does This Paper Address?
Face-to-face communication is a rich tapestry of speech and gesture, where gestures often provide crucial context and meaning. However, not all gestures are equally informative; many are simply extraneous hand movements that can confuse natural language processing (NLP) systems. The challenge lies in automatically distinguishing "salient" or meaningful gestures from irrelevant ones, and effectively integrating this information into NLP tasks. Existing methods often struggle with this by either concatenating all visual information or using separately trained models, failing to capture the dynamic and context-dependent nature of gesture's relevance. This paper aims to bridge this gap by learning to predict gesture salience directly from data, without requiring explicit gesture annotations.
Methods
The core innovation is conditional modality fusion (CMF), a framework that models gesture salience as a hidden variable. This means the model learns when gestures are important for a task (like coreference resolution) without being explicitly told.
Here's a breakdown:
- Hidden Variable for Salience: The model introduces a hidden variable that represents whether a gesture is "salient" for a given noun phrase pair. This variable is learned jointly with the coreference decision.
- Conditional Integration: Instead of always using gesture features, CMF selectively incorporates them based on the learned salience. This is achieved through different potential functions that define how gesture and verbal features interact.
- Models of Fusion:
- Same-Same: A baseline that concatenates all features (early fusion).
- Same-Zero: A model where gesture features are only used if the hidden variable indicates salience (gesture weights go to zero otherwise).
- Different-Zero: A more flexible model where verbal feature weights can also change based on gesture salience, reflecting how language might adapt when accompanied by meaningful gestures.
- Different-Different: A more complex model with separate weights for verbal and gestural features, depending on salience.
- Learning: The models are trained using a gradient-based optimization to maximize conditional log-likelihood, leveraging only coreference annotations.
Datasets
The research utilizes a newly collected multimodal corpus of presentations. This corpus consists of sixteen short videos from nine different speakers, totaling 1137 noun phrases. The videos feature speakers explaining mechanical devices using diagrams. This corpus differs from existing ones by focusing on spontaneous speech in a lecture/presentation style, where gestures are more frequent and direct, and by avoiding seated meeting scenarios.
Key Insights and Ideas
- Gesture Salience is Task-Dependent: The paper strongly suggests that the importance of a gesture is not inherent but rather tied to the specific language understanding task at hand. What's salient for coreference might not be for something else.
- Hidden Variables are Powerful: Treating gesture salience as a hidden variable allows the model to learn complex, context-dependent relationships, leading to more effective integration of multimodal information.
- Structured Fusion Outperforms Simple Concatenation: Simply combining all features (early fusion) is less effective than models that dynamically assess the relevance of gesture.
- Learned Salience Generalizes: The salience model learned for coreference resolution proved useful for a completely different task (keyframe extraction), suggesting a degree of generality in the learned concept of gesture salience.
Key Results
- Coreference Resolution: Conditional modality fusion (specifically the "different-zero" model) significantly outperformed early fusion and late fusion baselines, achieving an absolute improvement of 1.17% in AUC. This represents a 73% relative increase in the power of gesture features.
- Keyframe Extraction: When using the learned gesture salience to select keyframes for video summarization, the CMF approach significantly outperformed random keyframe selection, NP-salience, and pose-clustering baselines, achieving an F-Measure of .404. This demonstrates the practical utility of the learned salience model beyond coreference.
Limitations and Future Work
- Keyframe Selection Nuance: While CMF identified relevant noun phrases for keyframes, the exact keyframe selection (midpoint of the NP) was sometimes slightly off, especially for brief gestures. Future work could involve "looking inside" noun phrases to pinpoint the most salient gesture frame.
- Gesture Not Tied to Noun Phrases: The current approach only considers gestures associated with noun phrases. Gestures that are semantically important but not directly linked to an NP (e.g., a gesture for "up" not coinciding with an NP) are missed. Expanding the model to consider other phrase types or temporal patterns is suggested.
- Richer Models of Salience: The current salience model is binary (salient/not salient). Future research could explore more complex structural representations, such as Markov state models for temporal regularity or probabilistic context-free grammars for dependencies among gestures.
- Other Modalities: The paper highlights that hand gesture is just one of many coverbal modalities (facial expressions, body posture, etc.). Future work could explore the interplay and redundancy/complementarity of these modalities.
- Dataset Size: While the corpus is new and valuable, its size is noted as a limitation, particularly for parameter tuning and robust evaluation.
Summary updated