TL;DR
This paper tackles the challenge of retrieving relevant charts from a dataset using textual queries. The authors compare four multimodal retrieval approaches, finding that combining chart de-rendering with a text/table retrieval model (TAB-GTR) offers the best performance, especially when augmented with multi-task training. This work pioneers chart retrieval and highlights the complementary strengths of different modalities.
What Problem Does This Paper Address?
Retrieving specific information from visually rich documents like scientific papers or reports is a growing challenge. Charts, in particular, are crucial for summarizing complex data and conveying insights across various domains, from finance to scientific research. However, existing research on multimodal retrieval has largely overlooked chart retrieval, focusing more on text-image or text-table retrieval. This paper aims to bridge this gap by investigating and comparing different approaches for retrieving charts using textual queries, considering how to best represent and leverage the information contained within them.
Methods
The paper explores four distinct approaches for multimodal chart retrieval:
- OCR → Text Retrieval: Charts are first converted into text using Optical Character Recognition (OCR), and then a text retrieval model is used.
- Chart DeRendering → Table Retrieval: A specialized model (DEPLOT) converts the chart image into a structured table, which is then used for retrieval.
- VLM Retrieval: A Vision-Language Model (VLM), specifically PALI-3, directly processes the chart image for retrieval.
- Chart DeRendering → VLM Retrieval: This approach combines the de-rendered table from DEPLOT with the original chart image as input to the VLM.
A key contribution is TAB-GTR, a novel text retrieval model that incorporates table structure embeddings. This model is designed to improve performance on table retrieval tasks and serves as the backbone for the text/table-based approaches. The authors also investigate the impact of multi-task training, where models are trained on a mixture of data from different datasets and tasks, to improve robustness and generalization.
Datasets
The authors adapted several existing datasets for their chart retrieval task:
- NQ-TABLES: Used for evaluating the quality of the TAB-GTR model.
- CHARTQA (human & augmented): A chart question-answering dataset adapted for retrieval.
- CHART2TEXT (Statista & Pew): A chart summarization dataset adapted for retrieval.
- SCICAP: A chart captioning dataset adapted for retrieval.
These datasets were chosen due to the lack of dedicated chart retrieval benchmarks. The paper notes that some of these datasets contain contextual queries, which can add noise to retrieval metrics.
Key Insights and Ideas
- De-rendering charts into tables is powerful: For many chart types, converting them into structured tables (using DEPLOT) and then using a robust table retrieval model (TAB-GTR) significantly outperforms direct image understanding (PALI-3) when the data is in-distribution.
- TAB-GTR is a strong foundation: The proposed TAB-GTR model, which augments text retrieval with table structure embeddings, achieves state-of-the-art results on the NQ-TABLES benchmark and serves as an effective backbone for chart retrieval.
- Image-based models offer flexibility: While de-rendering can be highly effective, VLMs like PALI-3 demonstrate greater flexibility and can outperform de-rendering approaches on out-of-domain or complex charts, albeit at a much higher computational cost.
- Combining modalities is key: The most robust performance is achieved by combining chart de-rendering with VLM retrieval (PALI-3 + DEPLOT), suggesting that leveraging both structured table data and visual understanding is crucial for comprehensive chart retrieval.
- Multi-task training benefits image models: Multi-task learning appears to significantly benefit image-based models like PALI-3, likely by helping them learn a more generalized image representation. Text-based models, however, can sometimes suffer from task interference in multi-task settings.
- DEPLOT has limitations: The DEPLOT model struggles with complex charts (e.g., multiple subplots) and text-heavy charts, indicating a need for more robust de-rendering pipelines.
Key Results
- TAB-GTR achieved state-of-the-art results on the NQ-TABLES benchmark, improving R@1 by 4.4 points to 48.88%.
- On in-distribution data (like CHARTQA), the DEPLOT-based method (b) outperformed the direct image understanding model PALI-3 (c), while being significantly more efficient (300M vs 3B parameters).
- The combined PALI-3 + DEPLOT approach (d) achieved the strongest overall results across various datasets.
- For complex scientific charts (SCICAP), image-based models (PALI-3) were clear winners, outperforming de-rendering approaches.
- The OCR baseline generally performed the worst, except on text-heavy charts where it could be competitive.
Limitations and Future Work
- Dataset limitations: The chart retrieval datasets were adapted from other tasks (QA, summarization, captioning) and are limited to a few domains. Future work could expand to more diverse domains (e.g., finance, news).
- Deplotter robustness: The DEPLOT model showed limitations in generalizing to complex or out-of-domain charts. Future research should focus on improving the robustness and flexibility of chart de-rendering pipelines.
- Language focus: The study was limited to English. Exploring chart retrieval in other languages is a potential future direction.
- Model size and efficiency: While the combined approach yielded the best results, it involved larger models. Future work could explore more efficient ways to combine modalities.
- Multi-task learning challenges: Further investigation into mitigating task interference in multi-task learning for text-based models is warranted.
Summary updated