Modular Diffusion Models for Structured Visual Recognition

Siddhesh Khandelwal, Björn Ommer, Leonid SigalarXiv 2026 · arXiv

TL;DR

Modular Diffusion Models (MDMs) break down complex, structured visual recognition tasks into a sequence of simpler, task-specific conditional diffusion processes. By factorizing the joint distribution of heterogeneous outputs (like continuous bounding boxes and discrete class labels), MDMs preserve cross-component alignment, effectively capture scene ambiguities, and deliver strong probabilistic predictions without requiring task-specific architectures.

What Problem Does This Paper Address?

Most modern deep learning models for structured visual recognition—such as object detection, instance segmentation, and scene graph generation—are deterministic. They output a single "point estimate" for an image. However, real-world visual scenes are inherently ambiguous due to occlusions, poor lighting, or fuzzy boundaries. Deterministic models cannot express this uncertainty.

While generative diffusion models are excellent at capturing complex probability distributions, adapting them to structured prediction is notoriously difficult. A naive "joint diffusion" approach independently adds noise to boxes, labels, and masks, and tries to denoise them all at once. Unfortunately, this completely destroys the alignment between components during intermediate steps (e.g., a "person" mask gets paired with a "bicycle" label), creating chaotic states that the model struggles to fix.

Figure 1: Modular Diffusion Models factorize the complex joint structured distribution into several easy-to-estimate conditionals, each modeled using a diffusion process. In contrast to existing work (top), MDM (bottom) models a diffusion process over all components of the structured distribution. Additionally, compared to a naive joint approach (top), MDM preserves cross-component alignment during denoising, avoiding incoherent intermediate states and enabling more reliable reconstruction of structured outputs. In the example shown above, MDM is able to correctly identify and accurately segment the two zebras in the scene.

Methods

To solve the alignment issue, the authors introduce Modular Diffusion Models (MDMs). Instead of denoising everything simultaneously, MDM uses a Directed Acyclic Graph (DAG) to factorize the joint distribution into a chain of simpler, conditional diffusion modules.

For example, in instance segmentation, the pipeline is split into three sequential steps:

  1. Box Diffusion: Generates and refines spatial bounding boxes.
  2. Label Diffusion: Predicts categorical labels, conditioned on the refined boxes.
  3. Mask Diffusion: Generates pixel-level masks, conditioned on both the boxes and labels.
MDM factorization for the instance segmentation pipeline.

Key Technical Details:

  • Continuous Relaxation: To apply standard Gaussian diffusion to discrete elements (like class labels or binary masks), the authors relax them into continuous spaces using one-hot vectors and downsampled spatial grids.
  • Transformer-Based Denoising: Each module is instantiated using a transformer decoder. Input queries are constructed directly from noisy variables and dynamically modulated with time-aware ancestral conditioning (via adaLN-Zero).
  • Parameter Sharing: To keep the memory footprint low, the image encoder and transformer decoder weights are shared across all modules, using lightweight task-specific projection layers to alter behavior.
The transformer-based denoising architecture.

Datasets

The framework was evaluated on two widely accepted benchmark datasets:

  • MS-COCO 2017: Used for object detection and instance segmentation (118K training images, 5K validation images, spanning 80 object categories).
  • Visual Genome: Used for scene graph generation (108K images, 150 object categories, and 50 relation labels).

Key Insights and Ideas

  • Conditioning Prevents Chaos: By passing the denoised output of an upstream module (like boxes) as ancestral context to a downstream module (like labels), MDM maintains strict cross-component alignment throughout the denoising process.
  • Zero-Shot Query Scaling: Unlike deterministic models that operate on a fixed query budget, MDMs can scale to an arbitrary number of queries at inference time without any retraining. A model trained on 300 queries can be evaluated with 900 queries to boost accuracy.
  • Test-Time Biasing for Long-Tails: For tasks with severe class imbalances (like scene graphs), MDM allows researchers to adjust the trade-off between standard Recall and Mean Recall at inference time simply by tuning a logit-biasing parameter ($\lambda$), eliminating the need for expensive re-training.

Key Results

  • Instance Segmentation: On MS-COCO, MDM outperforms the diffusion-based baseline DiffusionInst by up to 8.4 $AP^{mask}$ and achieves parity with top-tier deterministic models like Mask DINO (scoring 45.9 $AP^{mask}$ and 50.7 $AP^{box}$).
  • Object Detection: MDM outperforms DiffusionDet by 3.9 AP when running both box and label diffusions, matching the competitive DINO-DETR baseline.
  • Scene Graph Generation: On Visual Genome, MDM beats the most competitive deterministic baselines by 2.6 / 1.6 points on Harmonic Recall (hR@50/100).
  • Uncertainty Modeling: Qualitatively, aggregating multiple samples from MDM yields intuitive heatmaps that clearly highlight areas of visual ambiguity, such as the exact boundaries of partially hidden objects.

Limitations and Future Work

  • Sequential Dependency Risk: Because MDM relies on a chain of conditional distributions, it is theoretically susceptible to error propagation—if the initial box diffusion fails completely, downstream label and mask modules will receive corrupted conditioning signals (though the authors show the model degrades gracefully under stress).
  • Orthogonal Architectural Enhancements: The current implementation relies on a simple transformer baseline. The authors note that integrating more sophisticated, state-of-the-art components (such as the hybrid encoders used in RT-DETR) represents a promising avenue for future performance gains.

Summary updated

Abstract

Traditional supervised methods for structured visual recognition tasks -- such as object detection, segmentation, and scene graph generation -- often produce deterministic, fixed outputs, limiting their ability to capture the inherent uncertainty in complex visual scenes. As a consequence, such point estimates are unable to capture the prediction uncertainty (or multi modality) intrinsic to these problems, often arising from natural ambiguities (e.g., ambiguity in size of partially occluded objects, local ambiguity of exact segmentation boundary, etc.) as well as noise and sparsity of training data. To address this limitation, we present Modular Diffusion Models (MDMs), a simple and novel framework that learns a distribution over structured outputs for a given input image. MDMs decompose the diffusion process into distinct, task-specific modules, each focused on capturing a different aspect of the structured information space, such as object categories, spatial locations, and inter-object relationships. This modular design allows each component to be learned independently, with seamless integration at inference without additional training. Furthermore, the modularity of MDMs enables the diffusion process to easily operate over the heterogeneous output space common in many structured learning tasks (e.g., a continuous bounding boxes and discrete class labels). Experimental results over three distinct structured tasks -- object detection, instance segmentation, and scene graph generation -- highlight the benefits of our proposed framework.
BibTeX
@misc{khandelwal2026modular,
  author = {Siddhesh Khandelwal and Björn Ommer and Leonid Sigal},
  title = {{Modular Diffusion Models for Structured Visual Recognition}},
  year = {2026},
  eprint = {2606.22702},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.22702},
}
Figure