Exploring Multimodal Language Models for Sustainability Disclosure Extraction: A Comparative Study

Tanay Gupta, Tushar Goel, Ishan VermaNAACL 2025

TL;DR

This paper investigates the effectiveness of Vision-Language Models (VLMs) in extracting sustainability information from complex, multimodal PDF reports. The study reveals that while VLMs show promise, no single model excels across all data formats, highlighting significant challenges in handling tables, charts, and infographics, and a tendency towards hallucination or incorrect extraction.

What Problem Does This Paper Address?

Organizations are increasingly publishing detailed sustainability reports to meet stakeholder demands for transparency. These reports, however, are often unstructured, relying heavily on text, infographics, and tables, making automated information extraction difficult. Traditional methods are time-consuming and prone to errors. While Large Language Models (LLMs) have shown success in text-based information extraction, the multimodal nature of these sustainability reports (combining text, images, and tables) necessitates exploring Vision-Language Models (VLMs) for more comprehensive extraction. This paper addresses the gap in understanding how well current VLMs can handle the diverse and complex formats found in these critical disclosures.

Methods

The researchers evaluated three prominent open-source VLMs: Llama 3.2 Vision-Instruct, Qwen2-VL-7B-Instruct, and LLaVA-v1.6-mistral-7b-hf. These models were chosen for their recent advancements and proven performance on related multimodal tasks. The core methodology involved feeding these VLMs with PDF page images and corresponding GRI (Global Reporting Initiative) disclosures, along with carefully crafted prompts designed to guide the extraction process. The extracted information was then validated by human experts, categorizing outputs as "Correct," "Incorrect," or "Hallucination." The study also introduced a novel approach for extracting GRI index tables using a combination of Table Transformer and Llama 3, which served as a benchmark for VLM performance on this specific task.

Datasets

The study utilized a dataset of approximately 700 English-language sustainability reports sourced from the SASB (Sustainability Accounting Standards Board) website. A subset of 380 reports was selected based on the presence of GRI index tables. For evaluation, a benchmark dataset was generated from these reports, focusing on GRI disclosures and their corresponding page numbers. The "No Data" scenario was also tested, where VLMs were presented with pages where the requested GRI disclosure was expected but absent.

Key Insights and Ideas

  • No One-Size-Fits-All VLM: The study strongly suggests that current VLMs have distinct strengths and weaknesses. No single model can efficiently handle all the multimodal data formats present in sustainability reports.
  • Textual Data is King (for some): Llama 3.2 Vision-Instruct performed best on text-heavy pages, demonstrating its proficiency in processing textual content.
  • Hallucination is a Major Hurdle: LLaVA, in particular, showed a significant tendency to hallucinate, generating information not present on the page, especially when dealing with complex layouts. This raises serious concerns about its reliability for critical data extraction.
  • "Incorrect" is More Common Than "Hallucination" for Some: Qwen2 exhibited a high proportion of "incorrect" predictions, often providing section headlines or disclosure titles instead of the specific requested information, indicating a struggle with fine-grained extraction.
  • Complex Layouts are a Bottleneck: Pages combining text, tables, and charts ("text + table," "chart + infographics," etc.) consistently challenged all VLMs, leading to accuracy drops below 50%.
  • Handling "No Data" is Crucial: A significant finding was the models' difficulty in gracefully handling missing information. Instead of stating "no data," they often attempted to provide an answer, leading to incorrect predictions. LLaVa showed a better ability to identify the absence of information in this scenario.
  • GRI Index Tables Require Specialized Tools: The custom approach of Table Transformer + Llama 3 significantly outperformed individual VLMs in extracting GRI index tables, highlighting the need for specialized tools for structured data within reports.

Key Results

  • Overall Performance Varies: Llama 3.2 Vision-Instruct and Qwen2 showed comparable overall performance, while LLaVA lagged behind.
  • Text-Heavy Pages: Llama 3.2 Vision-Instruct achieved accuracies in the high 60s for text-based pages, while LLaVA struggled with accuracies in the high 30s.
  • Complex Page Types: Accuracy for pages with combined formats like "text+table" or "chart+infographics" often fell below 50% for all tested VLMs.
  • GRI Index Table Extraction: The custom Table Transformer + Llama 3 approach achieved 93% accuracy, significantly outperforming Llama 3.2 Vision (83.5%) and Qwen2 VL Chat (53.5%).
  • "No Data" Scenario: LLaVa demonstrated the highest accuracy in identifying missing information, while Llama and Qwen2 showed a high proportion of incorrect predictions.

Limitations and Future Work

The authors acknowledge that the manual validation process limited the number of reports included in the experiments. They also note that the performance of VLMs is sensitive to the specific content and language within different GRI categories.

Future work suggested includes:

  • Integrating Model Strengths: Developing hybrid approaches that combine the best features of different VLMs.
  • Fine-tuning for Improved Performance: Further training models on domain-specific data to enhance accuracy.
  • Knowledge-Infused Prompts: Exploring more sophisticated prompting techniques to guide extraction.
  • Addressing "No Data" Cases: Developing better mechanisms for VLMs to recognize and respond to information gaps, rather than attempting to generate incorrect answers.

Summary updated

BibTeX
@inproceedings{gupta2025exploring,
  author = {Tanay Gupta and Tushar Goel and Ishan Verma},
  title = {{Exploring Multimodal Language Models for Sustainability Disclosure Extraction: A Comparative Study}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure