Abstract
DeSa2VA (Decoupled Semantic-Aware Visual Augmentation) improves video and image segmentation by "decoupling" the entangled hidden states of Multimodal Large Language Models (MLLMs) into distinct visual and textual feature streams. By training the Segment Anything Model 2 (SAM-2) to specifically interpret these separated cues and using a self-feedback refinement loop, the framework achieves state-of-the-art performance in both reasoning-based segmentation and visual question answering.
Introduction
Recent foundation models like SAM-2 excel at identifying object boundaries but lack inherent semantic reasoning—they do not "know" what they are looking at without an external prompt. Conversely, Multimodal Large Language Models (MLLMs) are adept at reasoning but produce high-dimensional hidden states that are difficult for segmentation models to digest directly.
Existing approaches typically use a single [SEG] token to bridge these two worlds. However, this often leads to insufficient semantics, modality gaps, and feature entanglement where dynamic visual data and static text labels create noise that degrades accuracy in complex scenes. DeSa2VA addresses these issues through a novel decoupling strategy.
The DeSa2VA Framework
The core innovation is the Multimodal Information Decoupler (MDS). Instead of a single fusion point, the authors use a dual-linear-layer architecture to split MLLM outputs into two parallel tracks: one for textual annotations and one for visual cues.
Disentanglement Strategies
To ensure these features are truly distinct, the framework employs two sophisticated training techniques:
- Adversarial Training: A "Modality Discrimination-Driven Disentanglement Game" uses min-max optimization to force textual and visual features into orthogonal subspaces. If a discriminator cannot distinguish the modality of a feature, the disentanglement is considered successful.
- Mutual Information (MI) Minimization: The Contrastive Log-ratio Upper Bound (CLUB) estimator is used to mathematically ensure that the two streams share minimal redundant information.
Training and Refinement
The authors introduce a Text-Visual Alignment Pre-training phase to teach SAM-2 to map semantic descriptions to geometric points. Finally, a Self-Feedback Generation loop is used, where a predicted mask is fed back into the model as a "dense prompt" to iteratively refine the object's edges.
Experimental Evaluation
The model was trained and evaluated on a massive scale of approximately 1.1 million pairs across image and video segmentation tasks (RefCOCO, MeVIS, Ref-DAVIS17) and question-answering benchmarks (SEED-Bench, Video-MME).
Key Results
DeSa2VA consistently outperformed the baseline Sa2VA and other state-of-the-art models:
- Image Segmentation: On RefCOCO, the 4B-parameter model achieved a score of 82.6, outperforming the baseline by 3.7 points.
- Video Segmentation: It reached 47.2 on MeVIS and 77.0 on Ref-DAVIS17, setting new benchmarks for reasoning-based video object segmentation.
- Reasoning and QA: In Grounded Caption Generation (GCG), it achieved a score of 28.7. Crucially, it maintained a high score of 73.3 on SEED-Bench, proving that segmentation capabilities do not compromise the LLM's reasoning "brain."
Discussion and Future Work
The "aha moment" of this research is that more information isn't always better if it is entangled. By forcing the model to separate "what the text says" from "what the pixels show," the decoder provides much more precise guidance.
One limitation noted is the "Video Interference" paradox, where temporal noise in video can slightly degrade static image performance. Future work will likely focus on temporal decoupling to handle the time dimension as a third decoupled stream, further preventing interference between static and dynamic tasks.
Summary updated