Masked Diffusion Captioning for Visual Feature Learning

Chao Feng, Zihao Wei, Andrew Owens

TL;DR

This paper introduces Masked Diffusion Captioning (MDC), a novel approach to learn visual features by training an image-conditioned masked diffusion language model to generate image captions. MDC offers a more robust and scalable way to learn visual representations compared to traditional autoregressive captioning, achieving competitive performance on various downstream vision tasks.

What Problem Does This Paper Address?

Learning effective visual representations is crucial for many computer vision tasks. While image captioning has been explored as a method for this, autoregressive approaches often suffer from a diminishing visual learning signal as the caption generation progresses. This means the image becomes less important for predicting later words in the caption. Existing solutions, like bidirectional generation or contrastive learning, have their own drawbacks. This paper aims to address this by proposing a new captioning paradigm that leverages masked diffusion language models (MDLMs) to provide a more consistent and position-independent visual supervision signal.

Methods

The core of this work is Masked Diffusion Captioning (MDC). It adapts the principles of Masked Diffusion Language Models (MDLMs) to the task of image captioning.

Here's how it works:

  • Image-Conditioned MDLMs: An image encoder processes an input image, and its features are used to condition a language model.
  • Masking and Reconstruction: During training, text tokens in an image-caption pair are randomly masked. The model is then trained to reconstruct these masked tokens, using both the visual features and the unmasked text tokens as context.
  • Diffusion Process: Unlike standard masked language models (like BERT), MDC uses a diffusion process. This involves a forward process that gradually masks tokens and a reverse process that learns to denoise and reconstruct the original caption. The masking ratio is determined by a time step t, which is sampled during training.
  • Position-Independent Supervision: A key advantage is that the visual supervision signal is not tied to token position. Each token contributes to the learning signal independently, mitigating the issue of diminishing visual influence seen in autoregressive models.
  • Sampling: For generating captions, MDC starts with a fully masked sequence and iteratively denoises it over a fixed number of steps, effectively reconstructing the caption.

Datasets

The paper uses several datasets for pretraining and evaluation:

  • Pretraining:
    • Conceptual 3M (CC3M)
    • Conceptual 12M (CC12M)
    • Recap-DataComp (a subset of DataComp)
  • Evaluation:
    • ImageNet-1K
    • Food101
    • CIFAR-10
    • CIFAR-100
    • Pets
    • MSCOCO
    • Flickr30k
    • ARO (for compositionality)
    • SugarCrepe (for compositionality)

Key Insights and Ideas

  • Unified Noise Schedule: MDC uses a time-based schedule for masking, which is more unified and less dataset-dependent than tuning fixed masking ratios for different datasets, as required by some BERT variants. This leads to more robust performance.
  • Scalability with Data: The approach shows promising scalability with the amount of pretraining data. More image-text pairs lead to improved linear probing performance, suggesting MDC can effectively leverage large-scale datasets.
  • Compositionality Boost: MDC demonstrates strong performance on vision-language compositionality tasks, outperforming CLIP and autoregressive captioning models. This suggests that the diffusion-based approach helps in learning more nuanced cross-modal relationships.
  • Descriptive Captions: When trained on richer textual descriptions, MDC can generate more descriptive captions, especially when longer output sequences are specified during sampling.

Key Results

  • Competitive Visual Features: MDC achieves performance comparable to autoregressive captioning and contrastive learning methods on linear probing tasks across various datasets and backbone architectures (ViT-B/32, ViT-B/16, ViT-L/14). For instance, on CC12M with ViT-B/16, MDC achieved an average accuracy of 77.1%, closely matching or exceeding other methods.
  • Improved Compositionality: On the ARO benchmark, MDC achieved 85.1% accuracy for relation attribute and 89.0% for coco order, outperforming CLIP (53.6% and 27.2% respectively) and ARC.
  • Reasonable Caption Generation: While not the primary focus, MDC can generate reasonable captions, with performance improving as the specified output sequence length increases (e.g., MDC-20 showing better results than MDC-10 on MSCOCO).
  • Ablation Studies: Removing the time step t from the loss function significantly degraded performance, highlighting its importance. Tuning the noise schedule (e.g., using [0.5, 1.0]) also improved performance on shorter captions.

Limitations and Future Work

  • Dataset Scale: The pretraining dataset size (around 10M pairs) and model sizes are considered academic scale. Scaling to larger datasets and models is a natural next step.
  • Content Bias: Training on datasets with undesirable content could lead to biased visual representations and malicious caption generation.
  • Fixed Sequence Length: Vanilla MDC generates captions of a fixed sequence length, requiring explicit specification during sampling. Future work could explore more flexible length generation.
  • FLOPs: The paper notes that MDC might use more FLOPs than autoregressive models, which could be a factor in its performance on certain metrics.
  • Captioning Capability: While MDC generates reasonable captions, the paper suggests further research is needed for a direct comparison of captioning capability between autoregression and masked diffusion, especially considering differences in how sequence length is handled.

Summary updated

Abstract

We learn visual features by captioning images with an image-conditioned masked diffusion language model, a formulation we call masked diffusion captioning (MDC). During training, text tokens in each image-caption pair are masked at a randomly chosen ratio, and a decoder conditioned on visual features is trained to reconstruct the original text. After training, the learned visual features can be applied to downstream vision tasks. Unlike autoregressive captioning, the strength of the visual learning signal in MDC does not depend on each token's position in the sequence, reducing the need for auxiliary objectives. Linear probing experiments across a variety of academic-scale models and datasets show that the learned visual features are competitive with those produced by autoregressive and contrastive approaches.
BibTeX
@inproceedings{feng2025masked,
  author = {Chao Feng and Zihao Wei and Andrew Owens},
  title = {{Masked Diffusion Captioning for Visual Feature Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure