Mamba Fusion: Learning Actions Through Questioning
Lists
Only you see this
Abstract
Video Language Models (VLMs) are crucial for generalizing across diverse tasks and using language cues to enhance learning. While transformer-based architectures have been the de facto in visionlanguage training, they face challenges like quadratic computational complexity, high GPU memory usage, and difficulty with long-term dependencies. To address these limitations, we introduce MambaVL, a novel model that leverages recent advancements in selective state space modality fusion to efficiently capture long-range dependencies and learn joint representations for vision and language data. MambaVL utilizes a shared state transition matrix across both modalities, allowing the model to capture a more comprehensive understanding of the actions in the scene. Furthermore, we propose a question-answering task that helps guide the model toward relevant cues. These questions provide critical information about actions, objects, and environmental context, leading to enhanced performance. As a result, MambaVL achieves state-of-the art performance in action recognition on the Epic-Kitchens-100 dataset and outperforms baseline methods in action anticipation. The code is available at https://github.com/Dongzhikang/MambaVL.
BibTeX
@inproceedings{beedu2024mamba,
author = {Apoorva Beedu and Zhikang Dong and Jason Sheinkopf and Irfan Essa},
title = {{Mamba Fusion: Learning Actions Through Questioning}},
booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
year = {2024},
doi = {10.1109/icassp49660.2025.10888933},
}Loading related content...