PerceptionDLM: Parallel Region Perception with Multimodal Diffusion Language Models

Yueyi Sun, Yuhao Wang, Jason Li, Ye Tian, Tao Zhang, Jacky Mai, Yihan Wang, Haochen Wang, Jinbin Bai, Ling Yang, Yunhai TongarXiv 2026 · arXiv

TL;DR

PerceptionDLM introduces a multimodal diffusion language model designed for efficient, parallel multi-region image captioning. By moving away from slow, sequential autoregressive decoding, it generates descriptions for multiple image regions simultaneously, achieving up to a 3.5× throughput speedup while maintaining highly competitive caption quality.

What Problem Does This Paper Address?

While Multimodal Large Language Models (MLLMs) excel at image-level comprehension, fine-grained, localized perception—such as accurately describing multiple specific regions within a single image—remains a major bottleneck.

Existing state-of-the-art models rely heavily on Autoregressive (AR) decoding. Under the AR paradigm, descriptions must be generated sequentially: the model captions one region token-by-token, then moves to the next. As the number of queried regions increases, inference cost and latency grow linearly, making dense-region perception difficult to scale for real-world applications.

Diffusion Language Models (DLMs) offer a non-autoregressive alternative with intrinsic token-level parallelism. However, directly extending them to localized vision tasks is challenging because standard architectures struggle to isolate features from different regions, leading to severe cross-region interference and hallucinations.

Methods

The authors propose PerceptionDLM, which is built upon a strong foundational baseline called PerceptionDLM-Base.

PerceptionDLM-Base

The base architecture integrates a pretrained SigLIP-2 vision encoder, a lightweight two-layer MLP connector, and a LLaDA-8B diffusion language model decoder. It incorporates a dynamic-resolution tiling strategy to handle high-resolution images without losing fine-grained spatial details.

Parallel Region Perception Architecture

To unlock true sequence- and token-level parallelism for multiple regions, the authors introduce three core architectural modifications:

  1. RoI-aligned Feature Replay: Localized visual features are extracted directly from the vision encoder using RoI-Align and projected into the language embedding space as placeholder tokens.
  2. Region Prompting: To help the model distinguish between concurrent targets, each region is assigned a learnable continuous embedding that acts as a region-specific visual prompt.
  3. Structured Attention Masking: This block-wise attention mechanism enforces region-wise independence while preserving global context. Tokens for a specific region are restricted to attending only to global visual tokens, shared textual prompts, their own RoI features, and their own caption span. Crucially, attention to other regions' features and captions is completely masked out.
Overview of the proposed parallel region perception architecture.

Datasets

Training Datasets

  • Base Pretraining: Trained across a 4-stage paradigm using Bee-Training-Data-Stage1 & Stage2, LLaVA-OneVision-1.5-Instruct-Data (22M samples), and Honey-Data-15M.
  • ParaCaption-5.7M: A new high-quality multi-mask caption dataset constructed by the authors. They repurposed the SA-1B and COCONut segmentation datasets, using an automated pipeline with a strong AR model (GAR-8B) and SAM3 to generate and verify dense, multi-region local captions.

Evaluation Benchmarks

  • Standard Benchmarks: 16 diverse multimodal benchmarks (including MMStar, MMBench, MMMU, MathVista, and ChartQA).
  • ParaDLC-Bench (Parallel Detailed Localized Captioning Benchmark): A new benchmark introduced in this paper to evaluate multi-mask scenarios. Scaled from the original DLC-Bench, it focuses on inter-region feature independence and anti-interference capability, utilizing GPT-5.2 as an LLM judge to evaluate both caption quality and inference efficiency.

Key Insights and Ideas

  • Exploiting DLM Parallelism Beyond Tokens: While standard DLMs parallelize token generation within a single sequence, this work demonstrates that they can parallelize generation across multiple independent sequences (captions) simultaneously if guided by proper structural masking.
  • The Necessity of Structural Isolation: Ablation studies revealed that replacing the structured attention mechanism with standard full attention caused a 6.2% drop in accuracy. Without block-wise isolation, the model suffers from severe "attribute entanglement," mistakenly blending features from neighboring objects.
  • Graceful Capacity Degradation: When forced into edge cases where the number of image masks exceeds the number of trained visual prompts, the model reuses prompts and experiences a graceful degradation in accuracy (dropping from 73.5% to 68.6%) rather than a catastrophic failure.

Key Results

  • State-of-the-Art Diffusion Baseline: PerceptionDLM-Base consistently outperforms prior diffusion VLMs, beating LLaDA-V on 15 out of 16 standard multimodal benchmarks.
  • Competitive Accuracy with Unprecedented Speed: On the ParaDLC-Bench, PerceptionDLM achieves an average accuracy of 62.4%, nearly doubling LLaDA-V (35.2%) and approaching strong AR region-specific models like GAR (69.5%).
  • Massive Efficiency Gains: While AR models are restricted to a Tokens Per Forward (TPF) metric of 1, PerceptionDLM achieves a TPF of 2.9. It slashed total benchmark inference time to just 276 seconds, compared to 479 seconds for GAR and 718 seconds for PixelRefer—representing up to a 3.5× throughput speedup in dense perception tasks.
Overview and efficiency analysis of PerceptionDLM.

Limitations and Future Work

  • Denoising Step Bottleneck: Although highly parallelized, the model's absolute inference speed remains bounded by the multi-step denoising process (defaulting to 32 steps) inherent to diffusion models. The authors suggest exploring step distillation techniques to compress generation into fewer steps.
  • Dense Attribute Entanglement: In extremely crowded or semantically similar scenes, the model still occasionally leaks attributes between adjacent masks. Incorporating explicit region contrastive losses could mitigate this.
  • The Reasoning Gap: Like many diffusion language models, PerceptionDLM faces a performance gap in complex, reasoning-heavy mathematical tasks compared to advanced AR models. The authors propose leveraging Reinforcement Learning (RL) pipelines to unlock deeper reasoning capabilities in future iterations.

Summary updated

Abstract

Multimodal large language models (MLLMs) have achieved remarkable progress in visual understanding tasks. However, most existing MLLMs rely on autoregressive generation, which limits their efficiency for perception tasks that require captioning multiple regions. In this work, we propose PerceptionDLM, a multimodal diffusion language model optimized for efficient parallel region perception. Built upon PerceptionDLM-Base, a strong foundational baseline that achieves state-of-the-art performance among open-source diffusion MLLMs, our architecture fully leverages the parallel decoding nature of DLMs. Specifically, we introduce efficient prompting and structured attention masking to enable simultaneous perception of multiple masked regions, allowing the model to generate region descriptions in parallel at both the sequence and token levels. This design significantly improves inference efficiency compared with existing approaches that process regions sequentially. To systematically evaluate the parallelism property of visual perception capability for DLMs, we construct a new Parallel Detailed Localized Captioning Benchmark (ParaDLC-Bench) by scaling the DLC-Bench to include multiple region masks per image, enabling joint evaluation of both caption quality and inference efficiency. Experiments demonstrate that PerceptionDLM maintains competitive performance in region captioning while achieving substantial speed improvements for multi-region perception tasks. Our results highlight the potential of multimodal diffusion language models for efficient, parallel visual perception. To the best of our knowledge, we are the first to achieve parallel region caption and perception by leveraging the advantages of diffusion language models. Code, models, and datasets are released.
BibTeX
@misc{sun2026perceptiondlm,
  author = {Yueyi Sun and Yuhao Wang and Jason Li and Ye Tian and Tao Zhang and Jacky Mai and Yihan Wang and Haochen Wang and Jinbin Bai and Ling Yang and Yunhai Tong},
  title = {{PerceptionDLM: Parallel Region Perception with Multimodal Diffusion Language Models}},
  year = {2026},
  eprint = {2606.19534},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.19534},
}
Figure