TL;DR
MULTIMON is an automated evaluation framework that discovers systematic failures in multimodal models by identifying "erroneous agreement"—cases where semantically distinct text prompts produce nearly identical embeddings. By leveraging large language models like GPT-4, MULTIMON categorizes these blind spots into natural-language descriptions and mass-produces thousands of novel adversarial prompts that expose vulnerabilities across state-of-the-art text-to-image, video, and 3D generation systems.
What Problem Does This Paper Address?
Text-guided multimodal systems (such as DALL-E, Midjourney, and Stable Diffusion) frequently fail in production in ways their creators did not anticipate. Standard evaluation relies heavily on human intuition to guess what might go wrong, creating a bottleneck that misses the "long tail" of model failures.
Furthermore, manually testing every possible combination of words is computationally prohibitive if evaluators have to run the full image or video generation decoder for every test case. The authors address this gap by creating an automated, systematic, and steerable evaluation framework that surfaces generalizable patterns of failure before a model is deployed.
Methods
The authors introduce MULTIMON (Multimodal Monitor), an automated pipeline that operates in three main phases:
- Scraping via Erroneous Agreement: Instead of generating expensive images, MULTIMON looks directly at the text embedding space (e.g., CLIP). It searches a text corpus for pairs of sentences that have very high CLIP cosine similarity but low semantic similarity according to a reference language model (DistillRoBERTA). If two sentences mean completely different things but look identical to CLIP, at least one of them will cause a generation failure.
- Categorizing with LLMs: A language model (like GPT-4) is prompted with these scraped failure pairs to identify general, systematic patterns of failure described in natural language (e.g., failing to process negation, quantifiers, or spatial relations).
- Generating New Failures: The LLM uses these natural language descriptions to generate hundreds of entirely new adversarial prompt pairs.
MULTIMON can also be "steered" toward specific domains (like self-driving cars or specific pop-culture topics) by filtering the initial text corpus with a domain classifier or appending instructions to the LLM generation prompt.
Datasets
The system scrapes initial text pairs from two widely used datasets:
- SNLI (Stanford Natural Language Inference)
- MS-COCO Captions
Rather than introducing a static new dataset benchmark, MULTIMON acts as a generative engine to dynamically discover and mass-produce new failure-inducing benchmarks.
Key Insights and Ideas
- The Embedding Bottleneck Trick: By evaluating failures at the text-encoder level (CLIP) rather than the image-decoder level, MULTIMON bypasses the massive computational cost of image/video generation during the discovery phase.
- Algorithmic Monoculture: Because many diverse downstream models share CLIP as their text-encoder backbone, a failure discovered in CLIP automatically transfers across different systems and even different modalities.
- Safety Filter Evasion: Adversaries can exploit systematic blind spots like negation to easily bypass commercial safety filters. For example, prompting "a person does not dismember himself" bypasses text filters looking for violence but still generates gory imagery because the underlying model ignores the negation.
Key Results
- Discovery of 14 Systematic Failures: MULTIMON successfully uncovered 14 distinct failure modes of the CLIP text-encoder, including systematic struggles with negation, numerical differences, spatial relations, and quantifiers.
- High Downstream Error Rates: Human annotators confirmed that downstream text-to-image models failed on 80.0% of the prompt pairs generated by MULTIMON, compared to just 20.5% for a random baseline.
- Cross-Modality Transfer: The failures successfully transferred beyond text-to-image models into text-to-video (VideoFusion) and text-to-3D (Shap-E) models. For instance, the prompt "a wind turbine at rest" erroneously generated a video of a spinning turbine.
Limitations and Future Work
- Incomplete Coverage: "Erroneous agreement" only catches failures where two different prompts map to the same embedding. It misses failures where a single prompt maps to a bad embedding or where the image decoder itself introduces errors.
- LLM Dependency: The quality of the systematic failure descriptions and generated instances heavily depends on the capabilities of the underlying LLM; GPT-4 outperformed GPT-3.5 significantly in categorization quality.
- Future Directions: The authors suggest extending this framework to automatically patch these vulnerabilities (e.g., via fine-tuning on the generated failures) and developing similar automated evaluation techniques for other complex domains like code or protein generation.
Summary updated