TL;DR
Modular Diffusion Models (MDMs) break down complex, structured visual recognition tasks into a sequence of simpler, task-specific conditional diffusion processes. By factorizing the joint distribution of heterogeneous outputs (like continuous bounding boxes and discrete class labels), MDMs preserve cross-component alignment, effectively capture scene ambiguities, and deliver strong probabilistic predictions without requiring task-specific architectures.
What Problem Does This Paper Address?
Most modern deep learning models for structured visual recognition—such as object detection, instance segmentation, and scene graph generation—are deterministic. They output a single "point estimate" for an image. However, real-world visual scenes are inherently ambiguous due to occlusions, poor lighting, or fuzzy boundaries. Deterministic models cannot express this uncertainty.
While generative diffusion models are excellent at capturing complex probability distributions, adapting them to structured prediction is notoriously difficult. A naive "joint diffusion" approach independently adds noise to boxes, labels, and masks, and tries to denoise them all at once. Unfortunately, this completely destroys the alignment between components during intermediate steps (e.g., a "person" mask gets paired with a "bicycle" label), creating chaotic states that the model struggles to fix.
Methods
To solve the alignment issue, the authors introduce Modular Diffusion Models (MDMs). Instead of denoising everything simultaneously, MDM uses a Directed Acyclic Graph (DAG) to factorize the joint distribution into a chain of simpler, conditional diffusion modules.
For example, in instance segmentation, the pipeline is split into three sequential steps:
- Box Diffusion: Generates and refines spatial bounding boxes.
- Label Diffusion: Predicts categorical labels, conditioned on the refined boxes.
- Mask Diffusion: Generates pixel-level masks, conditioned on both the boxes and labels.
Key Technical Details:
- Continuous Relaxation: To apply standard Gaussian diffusion to discrete elements (like class labels or binary masks), the authors relax them into continuous spaces using one-hot vectors and downsampled spatial grids.
- Transformer-Based Denoising: Each module is instantiated using a transformer decoder. Input queries are constructed directly from noisy variables and dynamically modulated with time-aware ancestral conditioning (via
adaLN-Zero). - Parameter Sharing: To keep the memory footprint low, the image encoder and transformer decoder weights are shared across all modules, using lightweight task-specific projection layers to alter behavior.
Datasets
The framework was evaluated on two widely accepted benchmark datasets:
- MS-COCO 2017: Used for object detection and instance segmentation (118K training images, 5K validation images, spanning 80 object categories).
- Visual Genome: Used for scene graph generation (108K images, 150 object categories, and 50 relation labels).
Key Insights and Ideas
- Conditioning Prevents Chaos: By passing the denoised output of an upstream module (like boxes) as ancestral context to a downstream module (like labels), MDM maintains strict cross-component alignment throughout the denoising process.
- Zero-Shot Query Scaling: Unlike deterministic models that operate on a fixed query budget, MDMs can scale to an arbitrary number of queries at inference time without any retraining. A model trained on 300 queries can be evaluated with 900 queries to boost accuracy.
- Test-Time Biasing for Long-Tails: For tasks with severe class imbalances (like scene graphs), MDM allows researchers to adjust the trade-off between standard Recall and Mean Recall at inference time simply by tuning a logit-biasing parameter ($\lambda$), eliminating the need for expensive re-training.
Key Results
- Instance Segmentation: On MS-COCO, MDM outperforms the diffusion-based baseline DiffusionInst by up to 8.4 $AP^{mask}$ and achieves parity with top-tier deterministic models like Mask DINO (scoring 45.9 $AP^{mask}$ and 50.7 $AP^{box}$).
- Object Detection: MDM outperforms DiffusionDet by 3.9 AP when running both box and label diffusions, matching the competitive DINO-DETR baseline.
- Scene Graph Generation: On Visual Genome, MDM beats the most competitive deterministic baselines by 2.6 / 1.6 points on Harmonic Recall (hR@50/100).
- Uncertainty Modeling: Qualitatively, aggregating multiple samples from MDM yields intuitive heatmaps that clearly highlight areas of visual ambiguity, such as the exact boundaries of partially hidden objects.
Limitations and Future Work
- Sequential Dependency Risk: Because MDM relies on a chain of conditional distributions, it is theoretically susceptible to error propagation—if the initial box diffusion fails completely, downstream label and mask modules will receive corrupted conditioning signals (though the authors show the model degrades gracefully under stress).
- Orthogonal Architectural Enhancements: The current implementation relies on a simple transformer baseline. The authors note that integrating more sophisticated, state-of-the-art components (such as the hybrid encoders used in RT-DETR) represents a promising avenue for future performance gains.
Summary updated