CAT+: Investigating and Enhancing Audio-Visual Understanding in Large Language Models
Lists
Only you see this
Abstract
Multimodal Large Language Models (MLLMs) have gained significant attention due to their rich internal implicit knowledge for cross-modal learning. Although advances in bringing audio-visuals into LLMs have resulted in boosts for a variety of Audio-Visual Question Answering (AVQA) tasks, they still face two crucial challenges: 1) audio-visual ambiguity, and 2) audio-visual hallucination. Existing MLLMs can respond to audio-visual content, yet sometimes fail to describe specific objects due to the ambiguity or hallucination of responses. To overcome the two aforementioned issues, we introduce the CAT+, which enhances MLLM to ensure more robust multimodal understanding. We first propose the Sequential Question-guided Module (SQM), which combines tiny transformer layers and cascades Q-Formers to realize a solid audio-visual grounding. After feature alignment and high-quality instruction tuning, we introduce Ambiguity Scoring Direct Preference Optimization (AS-DPO) to correct the problem of CAT+ bias toward ambiguous descriptions. To explore the hallucinatory deficits of MLLMs in dynamic audio-visual scenes, we build a new Audio-visual Hallucination Benchmark, named AVHbench. This benchmark detects the extent of MLLM’s hallucinations across three different protocols in the perceptual object, counting, and holistic description tasks. Extensive experiments across video-based understanding, open-ended, and close-ended AVQA demonstrate the superior performance of our method. The AVHbench is released at https://github.com/rikeilong/Bay-CAT.
BibTeX
@article{ye2025cat,
author = {Qilang Ye and Zitong Yu and Rui Shao and Yawen Cui and Xiangui Kang and Xin Liu and Philip Torr and Xiaochun Cao},
title = {{CAT+: Investigating and Enhancing Audio-Visual Understanding in Large Language Models}},
journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
year = {2025},
url = {https://openreview.net/forum?id=RFsQ1ZeyYP},
}Loading related content...