Abstract
We present MILS (Multimodal Iterative LLM Solver), a surprisingly simple, training-free approach to imbue multimodal capabilities into large language models (LLMs). Leveraging their innate ability to perform multi-step reasoning, MILS prompts the LLM to generate candidate outputs, each of which is scored and fed back iteratively to refine the solution. This enables various applications that typically require training specialized models on task-specific data. We establish a new state-of-the-art on emergent zero-shot image, video, and audio captioning. Furthermore, MILS seamlessly applies to media generation, discovering prompt rewrites to improve text-to-image generation and style transfer. As a gradient-free optimization approach, MILS can also invert multimodal embeddings into text, enabling applications like cross-modal arithmetic.
Introduction
Traditionally, giving an LLM vision or audio capabilities requires "multimodal alignment"—a process of training models on massive datasets of paired images and text (e.g., CLIP or LLaVA). This paradigm is computationally expensive, requires highly curated data, and often locks a model into specific tasks.
This paper introduces MILS (Multimodal Iterative LLM Solver), a framework that demonstrates that massive multimodal pre-training is not the only path to perception. Instead, MILS pairs a "blind" LLM with an off-the-shelf "perceptual" model, allowing them to communicate iteratively until they solve a multimodal task. By treating tasks like captioning or image editing as a search problem at inference time, MILS achieves state-of-the-art zero-shot performance without a single gradient update or a byte of new training data.
The MILS Framework: Reasoning as Optimization
The core of MILS is a simple, gradient-free optimization loop consisting of two distinct modules:
- The GENERATOR (The "Brain"): A standard LLM (such as Llama 3.1). Its role is to propose candidate text solutions—for example, five different captions for an image or five refined prompts to improve an image generator.
- The SCORER (The "Eyes/Ears"): An off-the-shelf multimodal model (such as SigLIP for images, ViCLIP for video, or ImageBind for audio). The Scorer does not generate text; it provides a scalar similarity score between the input media and the LLM’s text candidates.
The Iterative Loop
The process follows a "guess and check" cycle:
- The LLM generates a batch of candidates.
- The Scorer ranks these candidates based on their alignment with the media.
- The top candidates and their respective scores are fed back into the LLM as a prompt.
- The LLM uses this feedback to reason about why certain candidates scored higher and generates a refined batch.
This cycle typically repeats for 10–20 steps until the solution converges on a high-fidelity description or instruction.
Key Capabilities and Applications
Emergent Zero-Shot Captioning
MILS allows LLMs to describe images, videos, and audio clips they were never trained to see or hear. Because it relies on the LLM's general reasoning, the descriptions often capture nuances that standard models might miss.
Prompt Rewriting and Image Enhancement
MILS can act as an iterative prompt engineer for text-to-image (T2I) generators like FLUX. By refining a simple prompt into a highly descriptive one that maximizes the "aesthetic preference" score of the output, MILS significantly improves the quality and faithfulness of generated images.
Cross-Modal Arithmetic
Because MILS can "invert" any modality into text, it enables semantic math across different senses. For example, a user can combine an image of a boat with the audio of "birds chirping." MILS generates a text description that fuses both concepts, which can then be used to generate a new, multi-sensory scene.
Experimental Evaluation
Since MILS is training-free, all datasets were used exclusively for evaluation to test the framework's emergent capabilities.
- Image Captioning: On the MSCOCO dataset, MILS achieved a 33.3 CIDEr score, outperforming previous gradient-based zero-shot methods like ZeroCap and ConZIC.
- Video Captioning: On MSR-VTT, MILS outperformed models specifically trained on the massive HowTo100M dataset in METEOR scores, despite having no video-specific training.
- Human Preference: In T2I generation tasks using DrawBench, human annotators preferred MILS-enhanced images over base model outputs over 70% of the time for both visual quality and prompt faithfulness.
- Style Transfer: MILS successfully discovered "edit prompts" to apply complex styles (e.g., Picasso) to target images without specialized editing training.
Discussion and Limitations
While MILS offers a powerful alternative to traditional training, it faces two primary challenges:
- Inference Latency: The iterative nature of the loop (10–20 passes) is significantly slower than a single forward pass in a pre-trained multimodal model.
- Scorer Dependency: The system's performance is capped by the Scorer's perceptual limits. If the Scorer cannot distinguish fine-grained spatial relationships, the LLM cannot reason its way to a correct description.
Conclusion
MILS demonstrates that the reasoning capabilities of modern LLMs are robust enough to solve complex multimodal tasks through iterative optimization. By decoupling perception from reasoning, MILS provides a flexible, training-free path toward more general and adaptable AI systems.
Summary updated