Audio-Oriented Multimodal Machine Comprehension via Dynamic Inter- and Intra-modality Attention

Zhiqi Huang, Fenglin Liu, Xian Wu, Shen Ge, Helin Wang, Wei Fan, Yuexian Zou

Introduction

Traditional Machine Comprehension (MC) systems primarily process textual information to answer questions. However, human communication is inherently multimodal, often relying on prosody—tone, emphasis, and pauses—to convey meaning. This paper introduces the task of Audio-Oriented Multimodal Machine Comprehension (AOMMC), which requires models to synthesize both text and audio signals, mirroring real-world scenarios like English listening examinations (e.g., TOEFL).

Challenges in AOMMC

Transitioning from text-only MC to AOMMC presents two significant hurdles:

  1. The Modality Gap: Audio and text are heterogeneous data types. Finding fine-grained alignments between a continuous audio signal and discrete text tokens is complex.
  2. Unimodal Flexibility: Practical applications may not always provide both modalities. A robust system must maintain high performance even when only audio or only text is available at inference time.

The DIIA and MKD Framework

To address these challenges, the authors propose a dual-component architecture consisting of Dynamic Inter- and Intra-modality Attention (DIIA) and Multimodal Knowledge Distillation (MKD).

The DIIA and MKD architecture for multimodal machine comprehension.

Dynamic Inter- and Intra-modality Attention (DIIA)

The DIIA module facilitates deep fusion rather than simple concatenation. It employs Inter-modality Attention to align audio segments with corresponding text and Intra-modality Attention to capture relationships within a single source (e.g., text collocations). A conditional gating mechanism dynamically filters noise, ensuring the model focuses on the most salient features from both modalities.

Multimodal Knowledge Distillation (MKD)

The MKD module enables the model to handle missing modalities. During training, a multimodal "teacher" model distills its integrated knowledge into unimodal "student" modules. This allows the system to leverage cross-modal insights even when operating in a unimodal (text-only or audio-only) test environment.

Datasets: L-TOEFL and CET

The researchers introduced two new benchmarks: L-TOEFL (from official TOEFL materials) and CET (from the College English Test). These datasets are designed to be "trap-heavy," containing questions where the correct answer depends on nuances in the audio—such as sarcasm or emphasis—that might be ambiguous in the transcript alone.

Performance and Qualitative Analysis

The DIIA model demonstrates significant improvements, outperforming unimodal baselines by up to 21.08% in accuracy. Furthermore, the MKD module allows the model to exceed standard unimodal performance by up to 18.87% when tested on a single modality. Interestingly, the model showed a greater performance boost on conversations compared to lectures, likely due to the richer emotional and tonal shifts in dialogue.

Qualitative comparison showing how multimodal attention avoids unimodal "traps" by focusing on audio emphasis.

As shown in the visualization, the multimodal model correctly identifies the professor's decisive attitude by attending to emphatic audio cues (e.g., "Ever!") and specific text markers. In contrast, the unimodal model fails to capture this prosodic emphasis and selects an incorrect answer.

Conclusion

This work highlights the importance of audio as a "contextual enricher" in machine comprehension. By effectively bridging the modality gap through dynamic attention and preserving knowledge via distillation, the proposed framework sets a new standard for processing complex, multimodal communication.

Summary updated

Abstract

While Machine Comprehension (MC) has attracted extensive research interests in recent years, existing approaches mainly belong to the category of Machine Reading Comprehension task which mines textual inputs (paragraphs and questions) to predict the answers (choices or text spans). However, there are a lot of MC tasks that accept audio input in addition to the textual input, e.g. English listening comprehension test. In this paper, we target the problem of Audio-Oriented Multimodal Machine Comprehension, and its goal is to answer questions based on the given audio and textual information. To solve this problem, we propose a Dynamic Inter- and Intra-modality Attention (DIIA) model to effectively fuse the two modalities (audio and textual). DIIA can work as an independent component and thus be easily integrated into existing MC models. Moreover, we further develop a Multimodal Knowledge Distillation (MKD) module to enable our multimodal MC model to accurately predict the answers based only on either the text or the audio. As a result, the proposed approach can handle various tasks including: Audio-Oriented Multimodal Machine Comprehension, Machine Reading Comprehension and Machine Listening Comprehension, in a single model, making fair comparisons possible between our model and the existing unimodal MC models. Experimental results and analysis prove the effectiveness of the proposed approaches. First, the proposed DIIA boosts the baseline models by up to 21.08% in terms of accuracy; Second, under the unimodal scenarios, the MKD module allows our multimodal MC model to significantly outperform the unimodal models by up to 18.87%, which are trained and tested with only audio or textual data.
BibTeX
@inproceedings{huang2021audio,
  author = {Zhiqi Huang and Fenglin Liu and Xian Wu and Shen Ge and Helin Wang and Wei Fan and Yuexian Zou},
  title = {{Audio-Oriented Multimodal Machine Comprehension via Dynamic Inter- and Intra-modality Attention}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i14.17548},
}
Figure