Introduction
Audio-Visual Question Answering (AVQA) represents a significant challenge in multimodal machine learning, requiring models to synthesize information from video frames, audio tracks, and natural language queries. A primary obstacle in this field is information redundancy. In a typical video, the answer to a specific question—such as "Which instrument played first?"—may be contained within a mere fraction of the total temporal duration and a small spatial region of the visual field.
Standard AVQA models often process the entire video stream uniformly, which is not only computationally expensive but also introduces significant "noise" from irrelevant data. Furthermore, localizing the specific visual source of a sound without explicit bounding box supervision remains a difficult task. To address these challenges, researchers from Renmin University of China have introduced PSTP-Net, a framework designed to progressively filter irrelevant spatio-temporal data, focusing computational resources on the most informative segments of the input.
The PSTP-Net Framework
The Progressive Spatio-Temporal Perception Network (PSTP-Net) operates as a hierarchical funnel, discarding irrelevant information across three distinct stages to mimic human-like selective attention.
1. Temporal Segment Selection Module (TSSM)
The process begins by dividing the video into $K$ temporal segments. The TSSM utilizes cross-modal attention between the question and the combined audio-visual features to rank these segments. By selecting only the $Top_k$ most relevant time intervals, the model effectively ignores portions of the video that do not contribute to the answer, reducing the temporal search space.
2. Spatial Region Selection Module (SRSM)
Once the relevant time segments are identified, the SRSM focuses on the visual domain. Utilizing a CLIP-based backbone, the module identifies the $Top_m$ visual patches most relevant to the question. This step allows the model to ignore background scenery and static objects, focusing instead on the active "actors" or objects pertinent to the query.
3. Audio-guided Visual Attention Module (AVAM)
To resolve the spatial association between a sound and its source, the AVAM performs cross-modal attention between the audio stream and the previously selected visual patches. This stage is crucial for "sound-aware" perception, allowing the model to localize which visual object is responsible for a specific sound without requiring expensive human-labeled bounding boxes.
To ensure that the model does not lose the "big picture" necessary for global reasoning, the architecture includes a Lightweight Global Perception Module (LGPM). This module maintains a low-resolution summary of the entire video, which is fused with the high-resolution filtered features for the final answer prediction.
Experimental Evaluation
The PSTP-Net was evaluated on two primary benchmarks:
- MUSIC-AVQA: A dataset comprising approximately 9,300 videos of musical performances involving 22 instrument types, specifically designed for complex spatio-temporal reasoning.
- AVQA: A larger-scale dataset of approximately 57,000 videos covering daily activities, where answers often require simultaneous clues from both audio and visual modalities.
Performance and Efficiency
PSTP-Net demonstrated superior performance compared to existing state-of-the-art (SOTA) models, such as AVST, while maintaining a significantly smaller computational footprint.
- Accuracy: The model achieved 73.52% accuracy on MUSIC-AVQA (surpassing AVST's 71.59%) and 90.2% on the AVQA dataset.
- Efficiency: The most notable achievement is the model's efficiency. PSTP-Net reduced training parameters by 76.7% (from 18.48M to 4.30M) and computational costs (FLOPs) by 63.7% (from 3.19G to 1.22G).
- Robustness: Ablation studies highlighted the importance of the AVAM module, which improved performance on audio-visual localization questions by over 6%, validating its ability to link sounds to their visual sources effectively.
Conclusion and Future Work
The success of PSTP-Net suggests that in the context of high-dimensional video data, "less is more." By aggressively pruning the input space through progressive filtering, the model avoids the interference of irrelevant data, leading to higher accuracy and lower resource consumption.
While the model shows great promise, the researchers identified areas for future improvement. Currently, the model relies on a fixed frame sampling rate (1 fps), and exploring dynamic sampling could further optimize efficiency. Additionally, while the model excels at multi-modal reasoning, it occasionally struggles with purely audio-based questions where visual data acts as noise. Future iterations may explore the integration of Large Language Models (LLMs) to enhance the reasoning capabilities of the final prediction stage.
Summary updated