Mass-Producing Failures of Multimodal Systems with Language Models

Shengbang Tong, Erik Jones, Jacob Steinhardt

TL;DR

MULTIMON is an automated evaluation framework that discovers systematic failures in multimodal models by identifying "erroneous agreement"—cases where semantically distinct text prompts produce nearly identical embeddings. By leveraging large language models like GPT-4, MULTIMON categorizes these blind spots into natural-language descriptions and mass-produces thousands of novel adversarial prompts that expose vulnerabilities across state-of-the-art text-to-image, video, and 3D generation systems.

What Problem Does This Paper Address?

Text-guided multimodal systems (such as DALL-E, Midjourney, and Stable Diffusion) frequently fail in production in ways their creators did not anticipate. Standard evaluation relies heavily on human intuition to guess what might go wrong, creating a bottleneck that misses the "long tail" of model failures.

Furthermore, manually testing every possible combination of words is computationally prohibitive if evaluators have to run the full image or video generation decoder for every test case. The authors address this gap by creating an automated, systematic, and steerable evaluation framework that surfaces generalizable patterns of failure before a model is deployed.

Methods

The authors introduce MULTIMON (Multimodal Monitor), an automated pipeline that operates in three main phases:

  1. Scraping via Erroneous Agreement: Instead of generating expensive images, MULTIMON looks directly at the text embedding space (e.g., CLIP). It searches a text corpus for pairs of sentences that have very high CLIP cosine similarity but low semantic similarity according to a reference language model (DistillRoBERTA). If two sentences mean completely different things but look identical to CLIP, at least one of them will cause a generation failure.
  2. Categorizing with LLMs: A language model (like GPT-4) is prompted with these scraped failure pairs to identify general, systematic patterns of failure described in natural language (e.g., failing to process negation, quantifiers, or spatial relations).
  3. Generating New Failures: The LLM uses these natural language descriptions to generate hundreds of entirely new adversarial prompt pairs.

MULTIMON can also be "steered" toward specific domains (like self-driving cars or specific pop-culture topics) by filtering the initial text corpus with a domain classifier or appending instructions to the LLM generation prompt.

The MULTIMON pipeline: Scrape, Categorize, and Generate.

Datasets

The system scrapes initial text pairs from two widely used datasets:

  • SNLI (Stanford Natural Language Inference)
  • MS-COCO Captions

Rather than introducing a static new dataset benchmark, MULTIMON acts as a generative engine to dynamically discover and mass-produce new failure-inducing benchmarks.

Key Insights and Ideas

  • The Embedding Bottleneck Trick: By evaluating failures at the text-encoder level (CLIP) rather than the image-decoder level, MULTIMON bypasses the massive computational cost of image/video generation during the discovery phase.
  • Algorithmic Monoculture: Because many diverse downstream models share CLIP as their text-encoder backbone, a failure discovered in CLIP automatically transfers across different systems and even different modalities.
  • Safety Filter Evasion: Adversaries can exploit systematic blind spots like negation to easily bypass commercial safety filters. For example, prompting "a person does not dismember himself" bypasses text filters looking for violence but still generates gory imagery because the underlying model ignores the negation.

Key Results

  • Discovery of 14 Systematic Failures: MULTIMON successfully uncovered 14 distinct failure modes of the CLIP text-encoder, including systematic struggles with negation, numerical differences, spatial relations, and quantifiers.
  • High Downstream Error Rates: Human annotators confirmed that downstream text-to-image models failed on 80.0% of the prompt pairs generated by MULTIMON, compared to just 20.5% for a random baseline.
  • Cross-Modality Transfer: The failures successfully transferred beyond text-to-image models into text-to-video (VideoFusion) and text-to-3D (Shap-E) models. For instance, the prompt "a wind turbine at rest" erroneously generated a video of a spinning turbine.
Examples of inputs generated by MULTIMON causing identical errors across multiple state-of-the-art models.

Limitations and Future Work

  • Incomplete Coverage: "Erroneous agreement" only catches failures where two different prompts map to the same embedding. It misses failures where a single prompt maps to a bad embedding or where the image decoder itself introduces errors.
  • LLM Dependency: The quality of the systematic failure descriptions and generated instances heavily depends on the capabilities of the underlying LLM; GPT-4 outperformed GPT-3.5 significantly in categorization quality.
  • Future Directions: The authors suggest extending this framework to automatically patch these vulnerabilities (e.g., via fine-tuning on the generated failures) and developing similar automated evaluation techniques for other complex domains like code or protein generation.

Summary updated

Abstract

Deployed multimodal systems can fail in ways that evaluators did not anticipate. In order to find these failures before deployment, we introduce MultiMon, a system that automatically identifies systematic failures -- generalizable, natural-language descriptions of patterns of model failures. To uncover systematic failures, MultiMon scrapes a corpus for examples of erroneous agreement: inputs that produce the same output, but should not. It then prompts a language model (e.g., GPT-4) to find systematic patterns of failure and describe them in natural language. We use MultiMon to find 14 systematic failures (e.g., "ignores quantifiers") of the CLIP text-encoder, each comprising hundreds of distinct inputs (e.g., "a shelf with a few/many books"). Because CLIP is the backbone for most state-of-the-art multimodal systems, these inputs produce failures in Midjourney 5.1, DALL-E, VideoFusion, and others. MultiMon can also steer towards failures relevant to specific use cases, such as self-driving cars. We see MultiMon as a step towards evaluation that autonomously explores the long tail of potential system failures. Code for MULTIMON is available at https://github.com/tsb0601/MultiMon.
BibTeX
@inproceedings{tong2023mass,
  author = {Shengbang Tong and Erik Jones and Jacob Steinhardt},
  title = {{Mass-Producing Failures of Multimodal Systems with Language Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=T6iiOqsGOh},
}
Figure