VLG-BERT: Towards Better Interpretability in LLMs through Visual and Linguistic Grounding

Toufik Mechouma, Ismail Biskri, Serge RobertNAACL 2025

TL;DR

This paper introduces VLG-BERT, a novel Large Language Model (LLM) that enhances interpretability and meaning encoding by integrating visual and syntactic grounding. By leveraging real-world visual data and linguistic dependencies, VLG-BERT aims to create LLMs that better align with human understanding and produce more factually grounded outputs.

What Problem Does This Paper Address?

Large Language Models (LLMs) have demonstrated remarkable capabilities in various natural language processing tasks. However, they often suffer from "hallucination," producing plausible but factually incorrect outputs. This stems from their reliance on statistical patterns in training data rather than true semantic understanding. A key challenge is making these "black-box" models more interpretable and accountable, ensuring their decisions are explainable and grounded in verifiable facts. Furthermore, current LLMs primarily learn from textual data, lacking the rich, multi-sensory grounding that humans use to understand the world. This gap between purely textual representations and real-world semantics limits their ability to truly comprehend and generate meaningful language.

Methods

VLG-BERT builds upon the standard BERT architecture but introduces two key innovations:

  • Visual Grounding: Instead of random initialization for word embeddings, VLG-BERT uses latent representations extracted from a pre-trained Vision Transformer (ViT) trained on ImageNet. This vocabulary of "concrete words" is extended using WordNet's synonymy, hyponymy, and hypernymy relations to create a lookup table for initializing embeddings. This anchors word meanings to real-world visual concepts.
  • Syntactic Grounding: The model incorporates syntactic dependencies as a form of "ground truth." It uses an adjacency matrix to represent these dependencies and employs an Augmented Lagrangian Optimization method to enforce these syntactic constraints during the learning process. This involves customizing the prediction layer of BERT to predict an adjacency matrix that approximates the true syntactic structure of sentences. The Augmented Lagrangian method is chosen for its effectiveness in handling the non-convex optimization problems inherent in training deep learning models with constraints.

Datasets

  • Training: The English Wikipedia dump and BookCorpus were used for training. The Wikipedia dump provided 16 GB of plain text, and BookCorpus offered a substantial corpus of over 11,000 books.
  • Evaluation: The AG News dataset was used for text classification to evaluate the model's performance.

Key Insights and Ideas

  • Bridging the Vision-Language Gap: By integrating visual representations, VLG-BERT takes a significant step towards bridging the gap between language and vision, making LLMs more aligned with how humans perceive and understand the world.
  • Real-World Anchoring: Grounding word embeddings in real-world visual concepts (via ImageNet and WordNet) provides a stronger semantic foundation, potentially leading to more accurate and less hallucinatory outputs.
  • Syntactic Structure as Supervision: Using syntactic dependencies as ground truth for training the attention mechanism helps the model learn richer, more structured representations of language.
  • Interpretability through Constraints: The use of Augmented Lagrangian Optimization to enforce syntactic constraints offers a novel approach to improving the interpretability of LLMs by guiding their learning process towards linguistically meaningful structures.

Key Results

Experiments on text classification using the AG News dataset showed that VLG-BERT outperforms BERT Base across all metrics, including precision, recall, and F1-Scores for all classes. VLG-BERT achieved the highest accuracy at 97.56%. While VLG-BERT had a longer training time (122 hours) compared to BERT Base (96 hours on different hardware), this is attributed to the hardware configuration and highlights the efficiency gained from the learned embeddings.

Limitations and Future Work

  • Training Time: VLG-BERT experienced a longer training time than BERT Base, although this is acknowledged to be hardware-dependent.
  • Future Work: The authors suggest future comparisons with models like VisualBERT, LXMERT, and CLIP, particularly on multimodal tasks such as image captioning and visual question answering, to further demonstrate VLG-BERT's ability to integrate visual, syntactic, and semantic knowledge for a deeper understanding of multimodal interactions. They also plan to explore how these advancements can be applied to tasks like image captioning and visual question answering.

Summary updated

BibTeX
@inproceedings{mechouma2025vlg,
  author = {Toufik Mechouma and Ismail Biskri and Serge Robert},
  title = {{VLG-BERT: Towards Better Interpretability in LLMs through Visual and Linguistic Grounding}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure