TL;DR
This paper introduces MambaVL, a novel vision-language model that leverages Mamba's efficient state-space modeling for long-range dependencies and a unique question-answering paradigm to improve action recognition in egocentric videos. By framing action understanding as answering targeted questions about verbs and nouns, MambaVL achieves state-of-the-art results on the challenging Epic-Kitchens-100 dataset.
What Problem Does This Paper Address?
Understanding actions in videos, especially from an egocentric perspective (first-person view), is a significant challenge. Egocentric videos are characterized by rapid camera movements, limited fields of view, and frequent occlusions, making it difficult for models to capture temporal dynamics and contextual information. Traditional transformer-based vision-language models (VLMs) struggle with the quadratic complexity of attention mechanisms, hindering their ability to process long video sequences and learn long-term dependencies. While existing VLMs often rely on descriptive captions, this paper argues that a question-answering approach can more effectively guide models to focus on crucial cues for action understanding.
Methods
The core of MambaVL is its novel fusion mechanism based on Selective State Space Models (SSMs), inspired by the Mamba architecture. Unlike standard transformers, SSMs offer linear scaling with sequence length, making them efficient for long videos. MambaVL's key innovation is a shared state transition matrix (A) across both the vision and language modalities within the SSM. This shared matrix allows for cross-modal information exchange, enabling the model to learn joint representations and effectively select relevant information from both vision and language.
The model also introduces a new question-answering (QA) task for action recognition. Instead of relying solely on video captions, MambaVL generates two specific questions for each video: one focusing on the verb and another on the noun involved in the action. These questions are designed to elicit specific information about objects and actions, guiding the model's reasoning. The paper details a prompt-based method using GPT-4o to generate these questions, ensuring diversity and relevance.
The overall architecture involves:
- Video Encoder: Extracts features from video frames.
- Text Encoder: Extracts embeddings from text (questions).
- SSM-Fusion Module: This is the novel component where vision and language features are fused using shared SSMs. It processes the sequences and learns joint representations.
- Classification Head: A final layer for predicting the action (verb and noun).
Datasets
The primary dataset used for evaluation is Epic-Kitchens-100 (EK100). This dataset is specifically designed for egocentric video understanding and contains a large number of daily activities. The paper also mentions using pre-training data like K600, IN-21k, K400, SUN, and Ego4D for the visual backbones.
A new annotation framework for generating question-answer pairs for actions in EK100 is introduced. This involves rephrasing existing narrations with GPT-4o and then prompting it to generate two distinct questions (verb-focused and noun-focused) for each action.
Key Insights and Ideas
- Questioning as a Learning Signal: Framing action recognition as answering targeted questions (verb and noun) is more effective than relying on descriptive captions for egocentric videos. This approach encourages deeper reasoning and focuses the model on critical contextual cues.
- Shared SSM State Transition Matrix: The core innovation of MambaVL is the shared state transition matrix within the SSM for vision and language. This allows for efficient cross-modal fusion and learning of joint representations, overcoming limitations of traditional attention-based fusion.
- Mamba's Efficiency for Long Sequences: The adoption of Mamba's SSM architecture is crucial for handling the long temporal dependencies inherent in video data, offering linear scaling and reduced computational complexity compared to transformers.
- Flexibility of SSM Fusion: The proposed SSM-based fusion mechanism is presented as being flexible and adaptable to various input modalities, suggesting potential for broader applications beyond vision-language tasks.
Key Results
- State-of-the-Art Action Recognition: MambaVL achieves state-of-the-art performance on the Epic-Kitchens-100 dataset for action recognition, outperforming existing methods across verb, noun, and overall action classification.
- Improved Action Anticipation: The model also demonstrates strong performance in action anticipation, outperforming baselines like ORVIT.
- Efficiency Gains: The Mamba-based fusion block adds a negligible number of FLOPs and trainable parameters compared to other fusion methods, while significantly improving accuracy.
- Fusion Method Comparison: MambaVL's fusion method outperforms MLP and Transformer-based fusion methods by a significant margin (7-10%), highlighting its effectiveness in encoding long-range information and learning joint representations.
Limitations and Future Work
- Dependency on Question Generation: The performance of MambaVL for action anticipation is somewhat constrained by the quality of the generated questions, which rely on detected actions. Errors in action recognition can propagate to question generation.
- LLM Integration: The model does not train or fine-tune a large language model (LLM) itself but rather uses GPT-4o for question generation. This means its performance is indirectly tied to the capabilities of the external LLM.
- Future Directions: The authors suggest that their flexible fusion method opens new opportunities for Mamba's application in various cross-modal tasks. Further research could explore more direct integration of LLMs or investigate alternative question generation strategies.
Summary updated