TL;DR
This paper explores whether textual descriptions of images can imbue large language models (LLMs) with commonsense reasoning abilities, a known weakness in text-only models. By augmenting LLMs with image descriptions, the researchers show improved performance on commonsense machine reading comprehension tasks, suggesting visual information is a valuable, albeit indirect, source of commonsense knowledge.
What Problem Does This Paper Address?
Modern large language models (LLMs) like BERT have achieved remarkable success in natural language understanding tasks, leading to the assumption that they can acquire a vast amount of commonsense knowledge simply by training on massive text corpora. However, commonsense knowledge is often implicit and rarely explicitly stated in text, as people tend to omit obvious information when communicating. This "reporting bias" limits what LLMs can learn from text alone. Humans, on the other hand, acquire commonsense through direct experience and perception, with vision being a primary modality. This paper investigates whether textual descriptions of visual scenes, which capture aspects of human perception and experience, can serve as a viable alternative or supplement to text-only training for commonsense reasoning.
Methods
The core idea is to leverage image descriptions as an external knowledge source to enhance an LLM's commonsense reasoning capabilities. The authors frame this as a machine reading comprehension (MRC) task, specifically focusing on "script knowledge" – understanding everyday scenarios and their associated participants and events.
- Knowledge Extraction: They use the Visual Genome dataset, which provides dense captions for objects and regions within images. These descriptions are embedded using BERT-sentence embeddings and indexed for efficient retrieval.
- MRC Task Setup: They utilize the MCScript2.0 dataset, which consists of everyday scenarios described in text passages, paired with questions that require commonsense reasoning to answer.
- Visually Enhanced BERT: A pre-trained BERT model is fine-tuned on MCScript2.0. For questions that the baseline BERT model answers incorrectly (deemed "unanswerable questions"), the authors retrieve relevant region descriptions from Visual Genome based on the scenario of the passage. These descriptions are then prepended to the original text passage as additional input for a "Visually Enhanced BERT" model. This augmented input is then used to fine-tune the model again.
The retrieval process for the training set was manually curated to focus on the content of the descriptions rather than the retrieval mechanism itself. For evaluation (dev and test sets), the retrieval was also manual.
Datasets
- Visual Genome: A large-scale dataset of real-world images with dense captions for multiple objects and regions. Used as the source for image descriptions.
- MCScript2.0: A dataset of stories about everyday scenarios, each paired with questions requiring commonsense reasoning. Used for training and evaluating the MRC models.
Key Insights and Ideas
- Visual Descriptions as Commonsense Proxies: The paper strongly suggests that textual descriptions of visual scenes, even if not direct experiences, contain valuable commonsense knowledge that is difficult to extract from text alone due to reporting bias.
- Affordances and Location Knowledge: Image descriptions proved particularly beneficial for questions related to object affordances (what an object can be used for) and commonsense knowledge about object locations.
- Limitations of Abstract Concepts: While visual descriptions helped with concrete scenarios, they were less effective for questions requiring highly abstract commonsense reasoning or knowledge about concepts that are difficult to visually represent (e.g., "budget").
- Retrieval is Key (and Challenging): The success of the approach hinges on retrieving relevant image descriptions. While manual retrieval worked for evaluation, automating this process for large training sets is a significant future challenge.
Key Results
- The "Visually Enhanced BERT" model achieved improved accuracy compared to the baseline BERT model on the commonsense MRC task.
- On the Dev Commonsense set: Baseline BERT achieved 0.780 accuracy, while Visually Enhanced BERT reached 0.857.
- On the Test Commonsense set: Baseline BERT achieved 0.732 accuracy, while Visually Enhanced BERT reached 0.749.
- The improvement was most pronounced for questions that the baseline BERT model initially answered incorrectly, particularly those related to affordances and object locations.
- Qualitatively, the paper shows examples where image descriptions directly helped the model infer answers that were previously unanswerable.
Limitations and Future Work
- Manual Retrieval: The retrieval of image descriptions was done manually for the training set (limited to 225 questions) and for the dev/test sets. This is time-consuming and not scalable for large-scale training.
- Abstract Concepts: The approach struggled with questions requiring highly abstract commonsense knowledge or concepts that are difficult to visually represent.
- Future Work: The authors aim to automate the retrieval process and explore more sophisticated ways of integrating region descriptions into LLMs beyond simply prepending them to the input. They also plan to investigate better ways of using visual information for commonsense acquisition.
Summary updated