Unified Language Representation for Question Answering over Text, Tables, and Images

Bowen Yu, Cheng Fu, Haiyang Yu, Fei Huang, Yongbin Li

Abstract

When trying to answer complex questions, people often rely on multiple sources of information, such as visual, textual, and tabular data. Previous approaches to this problem have focused on designing input features or model structure in the multi-modal space, which is inflexible for cross-modal reasoning or data-efficient training. In this paper, we call for an alternative paradigm, which transforms the images and tables into unified language representations, so that we can simplify the task into a simpler textual QA problem that can be solved using three steps: retrieval, ranking, and generation, all within a language space. This idea takes advantage of the power of pre-trained language models and is implemented in a framework called Solar. Our experimental results show that Solar outperforms all existing methods by 10.6-32.3 pts on two datasets, MultimodalQA and MMCoQA, across ten different metrics. Additionally, Solar achieves the best performance on the WebQA leaderboard.

Introduction

Answering complex questions often requires synthesizing information from diverse sources: a paragraph of text, a structured data table, and a photograph. Traditionally, researchers have tackled this by building "multi-modal" models that process pixels, table cells, and text tokens simultaneously. However, this approach faces two major hurdles: data scarcity and reasoning silos. Training a model to understand the relationship between pixels and text requires massive datasets, and many existing systems use separate "specialist" models for each modality, which prevents effective cross-modal reasoning.

The Solar framework addresses these challenges by shifting the paradigm from multi-modal processing to unified language representation. By converting images and tables into natural language descriptions, Solar leverages the massive world knowledge of pre-trained language models to outperform specialized multi-modal architectures while significantly reducing storage requirements.

The Solar Framework

The core innovation of Solar is the translation of all data modalities into a "language space" before they are processed by the reasoning engine. This unification allows the model to treat multi-modal QA as a standard text-based problem.

Modality Textualization

  • Table Textualization: Tables are converted into fluent sentences using simple templates. For example, a row in a sports table might be converted to: "Row one’s race is Santa Derby, the track is Santa Park."
  • Image Textualization: To ensure no critical visual detail is lost, Solar employs a dual-strategy for images:
  • Global Representation: A pre-trained captioning model (BLIP) describes the overall scene and semantic context.
  • Local Representation: An object-attribute detection model (VinVL) identifies specific items and their properties, such as "racing brown horse" or "white large stadium."

The Retrieve-Rank-Generate Pipeline

Once the data is unified into text, Solar employs a three-stage pipeline:

  1. Retrieve: Using Dense Passage Retrieval (DPR), the system identifies the most relevant text snippets, table rows, or image descriptions from the knowledge base.
  2. Rank: A BERT-based ranker sorts these retrieved clues to identify the most likely "gold" evidence required to answer the question.
  3. Generate: A T5 encoder-decoder model processes the question alongside the top-ranked clues to generate a natural language answer.

Experimental Evaluation

The researchers evaluated Solar on three major benchmarks designed to test multi-modal reasoning:

  • WebQA: Focuses on reasoning over images and text snippets.
  • MultimodalQA (MMQA): Requires combining information across text, tables, and images.
  • MMCoQA: A conversational multi-modal QA benchmark where the system must account for dialogue history.

Key Results

Solar achieved state-of-the-art (SOTA) results across all tested benchmarks. It outperformed previous SOTA methods by 10.6 to 32.3 points across ten different metrics on the MultimodalQA and MMCoQA datasets. Furthermore, Solar reached the #1 spot on the official WebQA leaderboard without the use of model ensembling or complex post-processing.

Ablation studies revealed that "Global" image descriptions (captions) were more impactful than "Local" object tags. This suggests that capturing the semantic relationships between objects is more critical for reasoning than simply identifying the objects themselves.

Discussion and Insights

Solar demonstrates that language can serve as a high-density interface for multi-modal information. Compared to raw pixels, language is a highly compressed representation; Solar’s text-based approach reduced the storage footprint for image features from 78GB to just 186MB—a 99.7% reduction.

By converting images to text, Solar also allows the model to leverage the "world knowledge" inherent in Large Language Models (LLMs). For instance, an LLM can use its internal knowledge (e.g., knowing the location of a specific landmark) to supplement the visual descriptions provided by the textualization step. Finally, because all clues are in the same text format, the T5 model can use its cross-attention mechanism to naturally link facts across modalities, such as connecting a name in a table to a description of a person in an image.

Conclusion and Future Work

Solar proves that transforming multi-modal data into a unified language space is a powerful and efficient strategy for complex QA. While the framework is currently limited by the quality of the initial vision-to-text translation, it sets a new standard for performance and storage efficiency. Future work will explore tighter integration between textualization models and the reasoning engine to minimize information loss and expand the approach to other multi-modal tasks like navigation and image generation.

Summary updated

Abstract

When trying to answer complex questions, people often rely on multiple sources of information, such as visual, textual, and tabular data. Previous approaches to this problem have focused on designing input features or model structure in the multi-modal space, which is inflexible for cross-modal reasoning or data-efficient training. In this paper, we call for an alternative paradigm, which transforms the images and tables into unified language representations, so that we can simplify the task into a simpler textual QA problem that can be solved using three steps: retrieval, ranking, and generation, all within a language space. This idea takes advantage of the power of pre-trained language models and is implemented in a framework called Solar. Our experimental results show that Solar outperforms all existing methods by 10.6-32.3 pts on two datasets, MultimodalQA and MMCoQA, across ten different metrics. Additionally, Solar achieves the best performance on the WebQA leaderboard
BibTeX
@inproceedings{yu2023unified,
  author = {Bowen Yu and Cheng Fu and Haiyang Yu and Fei Huang and Yongbin Li},
  title = {{Unified Language Representation for Question Answering over Text, Tables, and Images}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=A9xTEcVd8H},
}
Figure