Abstract
Chain-of-Description (CoD) is a novel prompting strategy for Multi-Modal Large Language Models (MLLMs) that requires the model to generate a detailed textual description of an audio or visual input before answering a query. By forcing the model to "put what it understands into words" first, researchers achieved significant performance gains—up to a 5.3% improvement on hard-level vision tasks—demonstrating that explicit verbalization acts as a powerful bridge between raw sensory data and logical reasoning.
Introduction
While Multi-Modal Large Language Models (MLLMs) have become increasingly capable of processing images and audio, they often struggle with a "leap of logic." Standard prompting typically asks these models to jump directly from a complex multi-modal signal—such as a dense scientific diagram or a long audio clip—to a final answer. This direct approach often fails because the model may perceive the necessary details but fail to attend to them during the reasoning phase.
The Chain-of-Description (CoD) Method
To address this, the authors introduce Chain-of-Description (CoD) Prompting, a two-step inference process:
- Description Generation: The model is first prompted to provide a comprehensive textual representation of the input (e.g., "Please describe this image in detail").
- Query Response: The model then uses its own generated description as context to answer the specific user question.
This approach is conceptually similar to "Chain-of-Thought" (CoT) prompting. However, instead of decomposing a logical problem into steps, CoD decomposes the perception of the input into a structured textual foundation.
Experimental Setup and Datasets
The study evaluated CoD across two primary domains:
- Audio (LALMs): Evaluated using AIR-Bench-Chat, containing 2,200 samples across Speech, Sound, Music, and Mixed Audio.
- Vision (LVLMs): Evaluated using MMMU_Pro, a robust benchmark categorizing tasks into Easy, Medium, and Hard levels.
Key Insights
The core philosophy of CoD is that "what I can understand, I can put into words." The generated description serves as a sanity check for the model's internal state.
Interestingly, CoD acts as a "Hard Task" Specialist. While the detailed description can sometimes introduce "noise" for easy tasks, it provides essential "anchors" for complex reasoning. Furthermore, the researchers found that CoD is particularly effective for human speech due to its high information density—rich in textual content, emotions, and context.
Results
Quantitative results demonstrate the effectiveness of CoD across various models and difficulty levels. On the MMMU_Pro benchmark, CoD increased the accuracy of both Qwen2-VL and Qwen2.5-VL on "Hard" level tasks by 5.3%.
In the audio domain, using Qwen2-Audio, CoD improved speech category performance by nearly 4% on the AIR-Bench-Chat benchmark, with an average alignment increase of 1.79% across all categories.
Conclusion and Future Work
While CoD significantly improves reasoning accuracy, it introduces computational overhead due to increased latency and token consumption. Future work may involve integrating "description-first" tasks into the pre-training phase of MLLMs to bake this capability directly into the model's weights, potentially mitigating the inference-time cost.
Summary updated