Unified Language Representation for Question Answering over Text, Tables, and Images

Bowen Yu, Cheng Fu, Haiyang Yu, Fei Huang, Yongbin Li

Introduction

In the real world, answering complex questions often requires synthesizing information from diverse sources: a paragraph of history (text), a census chart (table), and a landmark photo (image). Current AI systems struggle with this "Multi-modal QA" due to inflexible modality silos—where separate models for text, tables, and images fail to communicate—and data inefficiency, requiring massive datasets to train joint vision-language models.

The researchers behind Solar propose a radical simplification: instead of building complex multi-modal architectures, why not translate all visual and tabular data into natural language? By unifying all data sources into a single "language space," Solar leverages the pre-existing reasoning power of large language models (LLMs), achieving state-of-the-art results while reducing storage requirements by over 99%.

The Solar Framework

Solar maps heterogeneous data into a Unified Language Space through a three-phase process:

  1. Unified Representation (Translation):
    • Tables: Rows and columns are converted into sentences using simple templates (e.g., "Row one's race is Santa Derby, the track is Santa Park").
  2. Images: To preserve detail, Solar generates both Global macro-level captions (using BLIP) and Local descriptions of specific objects and attributes (using VinVL).
  3. The QA Pipeline: Once data is unified as text, the system employs a standard retrieval-augmented generation (RAG) pipeline. It uses Dense Passage Retrieval (DPR) to find relevant "clues," a BERT-based ranker to prioritize them, and a T5 encoder-decoder model to generate the final answer.
The Solar pipeline converts multi-modal data into a unified text format for retrieval and generation.

Advantages of a Unified Language Space

The "aha moment" of Solar is treating language as a universal interface. LLMs have already ingested billions of words and possess immense world knowledge. By converting images and tables to text, we allow the LLM to apply its existing reasoning skills rather than training a new model from scratch on limited multi-modal data.

Furthermore, this approach offers extreme compression. Text is a high-density information carrier; converting image features to text descriptions reduced storage needs from 78GB to 186MB—a 99.7% reduction. This makes the system highly viable for low-latency, real-world applications. Finally, because clues from different modalities are just sentences in the same input string, the model's attention mechanism naturally performs cross-modal reasoning without specialized architectures.

Experimental Results

Solar was evaluated on three major benchmarks: WebQA (multi-hop reasoning), MultimodalQA (integration of text, tables, and images), and MMCoQA (conversational multi-modal QA).

Solar dominated the leaderboards:

  • WebQA: Achieved the #1 spot, outperforming previous state-of-the-art models like MuRAG.
  • MultimodalQA: Improved Exact Match (EM) scores by 11.0 points over existing methods.
  • MMCoQA: On this challenging conversational task, Solar saw a massive performance leap, improving EM and F1 scores by over 32 points.
Performance comparison on the MMCoQA benchmark, showing Solar's significant lead over previous models.

Notably, Solar achieved these gains without the massive multi-modal pre-training (often involving hundreds of millions of image-text pairs) required by its competitors.

Conclusion

While Solar demonstrates the power of text-based unification, challenges remain. The system is susceptible to "cascading errors" if the initial translation (captioning or object detection) is inaccurate. Additionally, some visual nuances, such as complex spatial relationships, may be lost during text conversion. Future work will likely explore the "sweet spot" where language models and multi-modal features can be integrated even more seamlessly.

Summary updated

Abstract

When trying to answer complex questions, people often rely on multiple sources of information, such as visual, textual, and tabular data. Previous approaches to this problem have focused on designing input features or model structure in the multi-modal space, which is inflexible for cross-modal reasoning or data-efficient training. In this paper, we call for an alternative paradigm, which transforms the images and tables into unified language representations, so that we can simplify the task into a simpler textual QA problem that can be solved using three steps: retrieval, ranking, and generation, all within a language space. This idea takes advantage of the power of pre-trained language models and is implemented in a framework called Solar. Our experimental results show that Solar outperforms all existing methods by 10.6-32.3 pts on two datasets, MultimodalQA and MMCoQA, across ten different metrics. Additionally, Solar achieves the best performance on the WebQA leaderboard
BibTeX
@inproceedings{yu2023unified,
  author = {Bowen Yu and Cheng Fu and Haiyang Yu and Fei Huang and Yongbin Li},
  title = {{Unified Language Representation for Question Answering over Text, Tables, and Images}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=A9xTEcVd8H},
}
Figure