Introduction
Traditional Machine Comprehension (MC) systems primarily process textual information to answer questions. However, human communication is inherently multimodal, often relying on prosody—tone, emphasis, and pauses—to convey meaning. This paper introduces the task of Audio-Oriented Multimodal Machine Comprehension (AOMMC), which requires models to synthesize both text and audio signals, mirroring real-world scenarios like English listening examinations (e.g., TOEFL).
Challenges in AOMMC
Transitioning from text-only MC to AOMMC presents two significant hurdles:
- The Modality Gap: Audio and text are heterogeneous data types. Finding fine-grained alignments between a continuous audio signal and discrete text tokens is complex.
- Unimodal Flexibility: Practical applications may not always provide both modalities. A robust system must maintain high performance even when only audio or only text is available at inference time.
The DIIA and MKD Framework
To address these challenges, the authors propose a dual-component architecture consisting of Dynamic Inter- and Intra-modality Attention (DIIA) and Multimodal Knowledge Distillation (MKD).
Dynamic Inter- and Intra-modality Attention (DIIA)
The DIIA module facilitates deep fusion rather than simple concatenation. It employs Inter-modality Attention to align audio segments with corresponding text and Intra-modality Attention to capture relationships within a single source (e.g., text collocations). A conditional gating mechanism dynamically filters noise, ensuring the model focuses on the most salient features from both modalities.
Multimodal Knowledge Distillation (MKD)
The MKD module enables the model to handle missing modalities. During training, a multimodal "teacher" model distills its integrated knowledge into unimodal "student" modules. This allows the system to leverage cross-modal insights even when operating in a unimodal (text-only or audio-only) test environment.
Datasets: L-TOEFL and CET
The researchers introduced two new benchmarks: L-TOEFL (from official TOEFL materials) and CET (from the College English Test). These datasets are designed to be "trap-heavy," containing questions where the correct answer depends on nuances in the audio—such as sarcasm or emphasis—that might be ambiguous in the transcript alone.
Performance and Qualitative Analysis
The DIIA model demonstrates significant improvements, outperforming unimodal baselines by up to 21.08% in accuracy. Furthermore, the MKD module allows the model to exceed standard unimodal performance by up to 18.87% when tested on a single modality. Interestingly, the model showed a greater performance boost on conversations compared to lectures, likely due to the richer emotional and tonal shifts in dialogue.
As shown in the visualization, the multimodal model correctly identifies the professor's decisive attitude by attending to emphatic audio cues (e.g., "Ever!") and specific text markers. In contrast, the unimodal model fails to capture this prosodic emphasis and selects an incorrect answer.
Conclusion
This work highlights the importance of audio as a "contextual enricher" in machine comprehension. By effectively bridging the modality gap through dynamic attention and preserving knowledge via distillation, the proposed framework sets a new standard for processing complex, multimodal communication.
Summary updated