TL;DR
MILS (Multimodal Iterative LLM Solver) is a training-free framework that grants Large Language Models (LLMs) multimodal capabilities by treating them as iterative optimizers. By pairing an LLM "Generator" with an off-the-shelf "Scorer" (like CLIP), MILS achieves state-of-the-art zero-shot performance in image, video, and audio captioning, and significantly improves text-to-image generation—all without a single gradient update or task-specific training example.
What Problem Does This Paper Address?
While LLMs are undisputed masters of text, imbuing them with "vision" or "hearing" typically requires massive, expensive training on paired multimodal datasets (e.g., millions of image-caption pairs). Even existing "zero-shot" models are often pre-trained on curated data for specific tasks, making them brittle when faced with new modalities or complex reasoning requirements.
The authors ask a compelling question: Can we leverage the innate reasoning and "test-time compute" abilities of LLMs to solve multimodal tasks without any new training? The challenge lies in bridging the gap between continuous multimodal embeddings (what a vision or audio model sees) and discrete text (what an LLM understands) in a way that is flexible, gradient-free, and modality-agnostic.
Methods: The MILS Framework
The core of the paper is MILS, a simple yet powerful iterative loop consisting of two key modules:
- The GENERATOR: A standard, pre-trained LLM (like Llama 3.1). Its job is to propose candidate solutions—such as potential captions or refined image-generation prompts—based on a task description and feedback from previous rounds.
- The SCORER: An off-the-shelf multimodal model (like CLIP, SigLIP, or ImageBind). It evaluates how well the Generator’s candidates match the input media (image, video, or audio) and provides a scalar score.
The Iterative Process: * The Generator produces a set of candidate text outputs. * The Scorer ranks these candidates against the target media. * The top-scoring candidates are fed back into the Generator as "feedback." * The LLM uses its reasoning ability to refine the candidates, repeating the process for $N$ steps until it converges on a high-quality solution.
Because MILS is gradient-free, it doesn't need access to the internal weights of the models. It treats the LLM as a "prompt rewriter" that optimizes entirely within the space of natural language.
Key Insights and Ideas
- LLMs as Natural Language Optimizers: The most striking insight is that LLMs can act as effective optimizers for non-textual tasks simply by iterating in text space. They don't need to "see" pixels if they can reason about the feedback provided by a Scorer.
- Emergent Zero-Shot vs. Trained Zero-Shot: Most "zero-shot" captioners are actually trained on captioning-like data. MILS is truly emergent; it uses a general-purpose LLM and a general-purpose similarity model to perform a task neither was specifically built for.
- Cross-Modal Arithmetic & Style Transfer: Because MILS can "invert" embeddings into text, it enables a form of multimodal algebra. For example, you can take the embedding of an image of a dog and the audio of birds chirping, use MILS to find a text description that satisfies both, and then feed that text into an image generator. It can even be used to edit prompts to achieve zero-shot style transfer.
- Modality Agnostic: The framework is "plug-and-play." To move from image to audio captioning, you simply swap a CLIP scorer for an ImageBind scorer.
Key Results
- State-of-the-Art Captioning: MILS outperformed existing zero-shot methods on MSCOCO image captioning, achieving a CIDEr score of 33.3 (compared to 26.0 for the previous best). It also set new benchmarks for zero-shot video and audio captioning.
- Better Image Generation: When used to "rewrite" prompts for models like FLUX.1, MILS dramatically improves compositional accuracy. Human annotators preferred MILS-enhanced images over the base model's output 75% of the time for quality and 79.5% for faithfulness to the original prompt.
- Scaling Matters: Performance scales positively with the size of the LLM. Larger models (e.g., Llama 3.1 70B) are significantly better at interpreting Scorer feedback to refine their guesses.
Limitations and Future Work
While MILS is highly versatile, it comes with trade-offs:
- Inference Latency: Because MILS is iterative (often requiring 10-20 steps), it is significantly slower than single-pass models. It explicitly trades "test-time compute" for accuracy.
- Generator Bottlenecks: The system is limited by the LLM's vocabulary and its ability to describe fine-grained textures or abstract styles that might be hard to articulate in text.
- Scorer Resolution: The quality of the output is strictly capped by the Scorer's ability to distinguish between candidates. If the Scorer can't tell the difference between two similar captions, the Generator cannot improve upon them.
Future directions include extending MILS to 3D assets and spatial reasoning tasks, as well as optimizing the iterative loop to require fewer steps for convergence.
Summary updated