DSRS: Teaching MLLMs to Perceive, Reason, and Segment in Complex Driving Scenes

Sicen Guo, Zhengfei Song, Kuanhsun Chen, Xu Huang, Ping Zhong, Wei Ye, Shaoyi Du, Rui FanarXiv 2026 · arXiv

TL;DR

DSRS-Net is a new reasoning segmentation framework designed specifically for the complex, high-stakes environment of autonomous driving. By integrating geometric depth priors through a novel hierarchical interaction strategy, it enables Multimodal Large Language Models (MLLMs) to move beyond simple object recognition toward human-like reasoning about traffic rules, spatial relations, and potential road hazards.

What Problem Does This Paper Address?

While Multimodal Large Language Models (MLLMs) have made huge strides in "reasoning segmentation"—the ability to segment an object based on complex, implicit text queries—they often stumble in driving scenarios. General-purpose models like LISA are great at identifying a "red fruit" but struggle with driving-specific nuances like "the vehicle in the opposing lane that is closer to us."

The authors identify a fundamental "modality gap": most MLLMs are pretrained almost exclusively on RGB images. In a driving context, visual texture alone isn't enough to infer distance, occlusion, or the subtle geometric cues needed to understand road layouts. Simply "plugging in" a depth map often fails because the model doesn't know how to align these new geometric features with its existing visual and linguistic knowledge.

Methods

The researchers propose DSRS-Net, which introduces two primary architectural innovations to bridge this gap:

  1. Geometric Prior Extractor (GPE): Instead of raw depth maps, this module uses ResNet blocks to extract structured geometric features from relative depth maps.
  2. Depth-Guided Hierarchical Feature Interaction (DHFI): This is the "brain" of the fusion process. It uses a stack of Depth-Aware Deformable Attention (DA²) blocks. As shown in the architecture diagram below, these blocks don't just concatenate features; they use geometric and positional affinity matrices to learn sampling offsets. This ensures that the model's attention is physically grounded in the scene's 3D structure.
An illustration of the DSRS-Net architecture, showing how RGB, depth, and text queries are fused before entering the LLM.

By projecting these geometric priors into the hierarchical vision feature space before they reach the Large Language Model (LLM), DSRS-Net learns to correlate the concept of a "car" or "lane" with both its appearance and its spatial positioning.

Datasets

The authors introduced the DSRS Dataset, a high-quality benchmark specifically for driving scene reasoning.

  • Scale: 1,561 image-instruction-mask samples.
  • Content: Images from real-world datasets (like nuScenes and Cityscapes) paired with relative depth maps and complex, multi-hop reasoning instructions.
  • Categories: The dataset covers four high-level domains: Spatial Relations, Potential Risks, Challenging Conditions, and Traffic Rules.

For broader validation, they also tested the model on standard benchmarks like RefCOCO/+/g and ReasonSeg.

Key Insights and Ideas

  • Alignment Over Addition: Simply adding a depth channel to an RGB image provides negligible gains. The "aha moment" here is that geometric features must be explicitly aligned with visual semantics through specialized attention mechanisms (like the DHFI) to be useful to an MLLM.
  • Geometric Grounding: By using depth-aware offsets, the model avoids "hallucinating" segments in flat areas of the image. It learns that a "lane" is a specific geometric structure on the ground plane, not just a gray texture.
  • Implicit vs. Explicit: The DSRS task forces models to handle implicit instructions (e.g., "Which object requires special attention?" in a scene with a jaywalking pedestrian). This moves perception closer to the decision-making logic required for actual driving.

Key Results

DSRS-Net significantly outperforms existing state-of-the-art (SoTA) models. On the DSRS benchmark, the InternVL2.5-8B version of the model achieved a gIoU of 71.2 and a cIoU of 79.7, a massive jump over the baseline LISA model (56.8 gIoU).

Quantitative comparison showing DSRS-Net outperforming other models across various representative scenarios.

Qualitatively, the difference is striking. In Figure 6, while other models might segment the wrong sign or fail to capture the full boundary of a traffic lane, DSRS-Net consistently identifies the correct target by reasoning through the scene's geometry.

Fig. 6. Qualitative comparisons on the DSRS dataset, where ground-truth areas are highlighted with white dashed boxes and wrong areas are highlighted with blue dashed circles.

Limitations and Future Work

Despite its success, DSRS-Net has a few "blind spots":

  • Extreme Conditions: In very low-light or heavy rain, both RGB visibility and depth estimation quality drop, leading to incomplete masks.
  • Small/Indistinct Targets: The model still struggles with very small objects like potholes or objects with fuzzy boundaries like sidewalks.
  • Contextual Confusion: Occasionally, the model gets tripped up by linguistic cues—for example, segmenting a rectangular window when asked for a "rectangular sign."

The authors suggest that future work should focus on uncertainty-aware adaptive fusion (letting the model decide when to trust depth vs. RGB) and evolving the framework from static reasoning to dynamic, sequential action planning.

Summary updated

Abstract

Existing reasoning segmentation frameworks typically employ multimodal large language models (MLLMs) with broad world knowledge to extract semantics in a human-like manner and generate corresponding segmentation masks. Nevertheless, these frameworks lack the necessary scene-specific awareness. This paper introduces driving scene reasoning segmentation (DSRS), a scene-specific task that requires models to interpret implicit instructions, infer spatial relations, detect potential risks, handle challenging conditions, and comprehend underlying traffic rules, ultimately generating accurate segmentation masks. Meanwhile, we establish a benchmark dataset to facilitate future research on the DSRS task. To pioneer this task, we propose DSRS-Net, a novel architecture with advanced reasoning segmentation capability for automated driving. Specifically, DSRS-Net seamlessly infuses geometric features for deeper reasoning segmentation, without compromising the original multimodal fusion mechanism. This strategy effectively mitigates the modality representation mismatch in MLLMs caused by RGB-only pretraining, while enhancing the integration of geometric, contextual, and linguistic features. Extensive experiments demonstrate the superior performance of DSRS-Net in both driving scene reasoning segmentation and other language-driven segmentation tasks across various public datasets. The source code and dataset will be released upon publication.
BibTeX
@misc{guo2026dsrs,
  author = {Sicen Guo and Zhengfei Song and Kuanhsun Chen and Xu Huang and Ping Zhong and Wei Ye and Shaoyi Du and Rui Fan},
  title = {{DSRS: Teaching MLLMs to Perceive, Reason, and Segment in Complex Driving Scenes}},
  year = {2026},
  eprint = {2611.06160},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06160},
}
Figure