Decoupled Seg Tokens Make Stronger Reasoning Video Segmenter and Grounder

Jisheng Dang, Wu Xudong, Bimei Wang, Lv Ning, Chen Jiayu, Jingwen Zhao, Yichu liu, Jizhao Liu, Juncheng Li, Teng WangarXiv 2025 · DOI · arXiv

Abstract

DeSa2VA (Decoupled Semantic-Aware Visual Augmentation) improves video and image segmentation by "decoupling" the entangled hidden states of Multimodal Large Language Models (MLLMs) into distinct visual and textual feature streams. By training the Segment Anything Model 2 (SAM-2) to specifically interpret these separated cues and using a self-feedback refinement loop, the framework achieves state-of-the-art performance in both reasoning-based segmentation and visual question answering.

Introduction

Recent foundation models like SAM-2 excel at identifying object boundaries but lack inherent semantic reasoning—they do not "know" what they are looking at without an external prompt. Conversely, Multimodal Large Language Models (MLLMs) are adept at reasoning but produce high-dimensional hidden states that are difficult for segmentation models to digest directly.

Comparison of existing methods versus the proposed DeSa2VA framework, highlighting the dual decoding and mask refinement stages.

Existing approaches typically use a single [SEG] token to bridge these two worlds. However, this often leads to insufficient semantics, modality gaps, and feature entanglement where dynamic visual data and static text labels create noise that degrades accuracy in complex scenes. DeSa2VA addresses these issues through a novel decoupling strategy.

The DeSa2VA Framework

The core innovation is the Multimodal Information Decoupler (MDS). Instead of a single fusion point, the authors use a dual-linear-layer architecture to split MLLM outputs into two parallel tracks: one for textual annotations and one for visual cues.

Detailed architecture of the Modality-Decoupled Segmentation (MDS) pipeline, showing the flow from encoders to the decoupled SAM2-Decoders.

Disentanglement Strategies

To ensure these features are truly distinct, the framework employs two sophisticated training techniques:

  • Adversarial Training: A "Modality Discrimination-Driven Disentanglement Game" uses min-max optimization to force textual and visual features into orthogonal subspaces. If a discriminator cannot distinguish the modality of a feature, the disentanglement is considered successful.
  • Mutual Information (MI) Minimization: The Contrastive Log-ratio Upper Bound (CLUB) estimator is used to mathematically ensure that the two streams share minimal redundant information.

Training and Refinement

The authors introduce a Text-Visual Alignment Pre-training phase to teach SAM-2 to map semantic descriptions to geometric points. Finally, a Self-Feedback Generation loop is used, where a predicted mask is fed back into the model as a "dense prompt" to iteratively refine the object's edges.

Experimental Evaluation

The model was trained and evaluated on a massive scale of approximately 1.1 million pairs across image and video segmentation tasks (RefCOCO, MeVIS, Ref-DAVIS17) and question-answering benchmarks (SEED-Bench, Video-MME).

Key Results

DeSa2VA consistently outperformed the baseline Sa2VA and other state-of-the-art models:

  • Image Segmentation: On RefCOCO, the 4B-parameter model achieved a score of 82.6, outperforming the baseline by 3.7 points.
  • Video Segmentation: It reached 47.2 on MeVIS and 77.0 on Ref-DAVIS17, setting new benchmarks for reasoning-based video object segmentation.
  • Reasoning and QA: In Grounded Caption Generation (GCG), it achieved a score of 28.7. Crucially, it maintained a high score of 73.3 on SEED-Bench, proving that segmentation capabilities do not compromise the LLM's reasoning "brain."

Discussion and Future Work

The "aha moment" of this research is that more information isn't always better if it is entangled. By forcing the model to separate "what the text says" from "what the pixels show," the decoder provides much more precise guidance.

One limitation noted is the "Video Interference" paradox, where temporal noise in video can slightly degrade static image performance. Future work will likely focus on temporal decoupling to handle the time dimension as a third decoupled stream, further preventing interference between static and dynamic tasks.

Summary updated

BibTeX
@misc{dang2025decoupled,
  author = {Jisheng Dang and Wu Xudong and Bimei Wang and Lv Ning and Chen Jiayu and Jingwen Zhao and Yichu liu and Jizhao Liu and Juncheng Li and Teng Wang},
  title = {{Decoupled Seg Tokens Make Stronger Reasoning Video Segmenter and Grounder}},
  year = {2025},
  eprint = {2506.22880},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.22880},
}
Figure