Introduction
In the real world, answering complex questions often requires synthesizing information from diverse sources: a paragraph of history (text), a census chart (table), and a landmark photo (image). Current AI systems struggle with this "Multi-modal QA" due to inflexible modality silos—where separate models for text, tables, and images fail to communicate—and data inefficiency, requiring massive datasets to train joint vision-language models.
The researchers behind Solar propose a radical simplification: instead of building complex multi-modal architectures, why not translate all visual and tabular data into natural language? By unifying all data sources into a single "language space," Solar leverages the pre-existing reasoning power of large language models (LLMs), achieving state-of-the-art results while reducing storage requirements by over 99%.
The Solar Framework
Solar maps heterogeneous data into a Unified Language Space through a three-phase process:
- Unified Representation (Translation):
- Tables: Rows and columns are converted into sentences using simple templates (e.g., "Row one's race is Santa Derby, the track is Santa Park").
- Images: To preserve detail, Solar generates both Global macro-level captions (using BLIP) and Local descriptions of specific objects and attributes (using VinVL).
- The QA Pipeline: Once data is unified as text, the system employs a standard retrieval-augmented generation (RAG) pipeline. It uses Dense Passage Retrieval (DPR) to find relevant "clues," a BERT-based ranker to prioritize them, and a T5 encoder-decoder model to generate the final answer.
Advantages of a Unified Language Space
The "aha moment" of Solar is treating language as a universal interface. LLMs have already ingested billions of words and possess immense world knowledge. By converting images and tables to text, we allow the LLM to apply its existing reasoning skills rather than training a new model from scratch on limited multi-modal data.
Furthermore, this approach offers extreme compression. Text is a high-density information carrier; converting image features to text descriptions reduced storage needs from 78GB to 186MB—a 99.7% reduction. This makes the system highly viable for low-latency, real-world applications. Finally, because clues from different modalities are just sentences in the same input string, the model's attention mechanism naturally performs cross-modal reasoning without specialized architectures.
Experimental Results
Solar was evaluated on three major benchmarks: WebQA (multi-hop reasoning), MultimodalQA (integration of text, tables, and images), and MMCoQA (conversational multi-modal QA).
Solar dominated the leaderboards:
- WebQA: Achieved the #1 spot, outperforming previous state-of-the-art models like MuRAG.
- MultimodalQA: Improved Exact Match (EM) scores by 11.0 points over existing methods.
- MMCoQA: On this challenging conversational task, Solar saw a massive performance leap, improving EM and F1 scores by over 32 points.
Notably, Solar achieved these gains without the massive multi-modal pre-training (often involving hundreds of millions of image-text pairs) required by its competitors.
Conclusion
While Solar demonstrates the power of text-based unification, challenges remain. The system is susceptible to "cascading errors" if the initial translation (captioning or object detection) is inaccurate. Additionally, some visual nuances, such as complex spatial relationships, may be lost during text conversion. Future work will likely explore the "sweet spot" where language models and multi-modal features can be integrated even more seamlessly.
Summary updated