TL;DR
PerceptionDLM introduces a multimodal diffusion language model designed for efficient, parallel multi-region image captioning. By moving away from slow, sequential autoregressive decoding, it generates descriptions for multiple image regions simultaneously, achieving up to a 3.5× throughput speedup while maintaining highly competitive caption quality.
What Problem Does This Paper Address?
While Multimodal Large Language Models (MLLMs) excel at image-level comprehension, fine-grained, localized perception—such as accurately describing multiple specific regions within a single image—remains a major bottleneck.
Existing state-of-the-art models rely heavily on Autoregressive (AR) decoding. Under the AR paradigm, descriptions must be generated sequentially: the model captions one region token-by-token, then moves to the next. As the number of queried regions increases, inference cost and latency grow linearly, making dense-region perception difficult to scale for real-world applications.
Diffusion Language Models (DLMs) offer a non-autoregressive alternative with intrinsic token-level parallelism. However, directly extending them to localized vision tasks is challenging because standard architectures struggle to isolate features from different regions, leading to severe cross-region interference and hallucinations.
Methods
The authors propose PerceptionDLM, which is built upon a strong foundational baseline called PerceptionDLM-Base.
PerceptionDLM-Base
The base architecture integrates a pretrained SigLIP-2 vision encoder, a lightweight two-layer MLP connector, and a LLaDA-8B diffusion language model decoder. It incorporates a dynamic-resolution tiling strategy to handle high-resolution images without losing fine-grained spatial details.
Parallel Region Perception Architecture
To unlock true sequence- and token-level parallelism for multiple regions, the authors introduce three core architectural modifications:
- RoI-aligned Feature Replay: Localized visual features are extracted directly from the vision encoder using RoI-Align and projected into the language embedding space as placeholder tokens.
- Region Prompting: To help the model distinguish between concurrent targets, each region is assigned a learnable continuous embedding that acts as a region-specific visual prompt.
- Structured Attention Masking: This block-wise attention mechanism enforces region-wise independence while preserving global context. Tokens for a specific region are restricted to attending only to global visual tokens, shared textual prompts, their own RoI features, and their own caption span. Crucially, attention to other regions' features and captions is completely masked out.
Datasets
Training Datasets
- Base Pretraining: Trained across a 4-stage paradigm using
Bee-Training-Data-Stage1&Stage2,LLaVA-OneVision-1.5-Instruct-Data(22M samples), andHoney-Data-15M. - ParaCaption-5.7M: A new high-quality multi-mask caption dataset constructed by the authors. They repurposed the SA-1B and COCONut segmentation datasets, using an automated pipeline with a strong AR model (GAR-8B) and SAM3 to generate and verify dense, multi-region local captions.
Evaluation Benchmarks
- Standard Benchmarks: 16 diverse multimodal benchmarks (including MMStar, MMBench, MMMU, MathVista, and ChartQA).
- ParaDLC-Bench (Parallel Detailed Localized Captioning Benchmark): A new benchmark introduced in this paper to evaluate multi-mask scenarios. Scaled from the original DLC-Bench, it focuses on inter-region feature independence and anti-interference capability, utilizing GPT-5.2 as an LLM judge to evaluate both caption quality and inference efficiency.
Key Insights and Ideas
- Exploiting DLM Parallelism Beyond Tokens: While standard DLMs parallelize token generation within a single sequence, this work demonstrates that they can parallelize generation across multiple independent sequences (captions) simultaneously if guided by proper structural masking.
- The Necessity of Structural Isolation: Ablation studies revealed that replacing the structured attention mechanism with standard full attention caused a 6.2% drop in accuracy. Without block-wise isolation, the model suffers from severe "attribute entanglement," mistakenly blending features from neighboring objects.
- Graceful Capacity Degradation: When forced into edge cases where the number of image masks exceeds the number of trained visual prompts, the model reuses prompts and experiences a graceful degradation in accuracy (dropping from 73.5% to 68.6%) rather than a catastrophic failure.
Key Results
- State-of-the-Art Diffusion Baseline: PerceptionDLM-Base consistently outperforms prior diffusion VLMs, beating LLaDA-V on 15 out of 16 standard multimodal benchmarks.
- Competitive Accuracy with Unprecedented Speed: On the ParaDLC-Bench, PerceptionDLM achieves an average accuracy of 62.4%, nearly doubling LLaDA-V (35.2%) and approaching strong AR region-specific models like GAR (69.5%).
- Massive Efficiency Gains: While AR models are restricted to a Tokens Per Forward (TPF) metric of 1, PerceptionDLM achieves a TPF of 2.9. It slashed total benchmark inference time to just 276 seconds, compared to 479 seconds for GAR and 718 seconds for PixelRefer—representing up to a 3.5× throughput speedup in dense perception tasks.
Limitations and Future Work
- Denoising Step Bottleneck: Although highly parallelized, the model's absolute inference speed remains bounded by the multi-step denoising process (defaulting to 32 steps) inherent to diffusion models. The authors suggest exploring step distillation techniques to compress generation into fewer steps.
- Dense Attribute Entanglement: In extremely crowded or semantically similar scenes, the model still occasionally leaks attributes between adjacent masks. Incorporating explicit region contrastive losses could mitigate this.
- The Reasoning Gap: Like many diffusion language models, PerceptionDLM faces a performance gap in complex, reasoning-heavy mathematical tasks compared to advanced AR models. The authors propose leveraging Reinforcement Learning (RL) pipelines to unlock deeper reasoning capabilities in future iterations.
Summary updated