Abstract
Language models trained on large text corpora encode rich distributional information about real-world environments and action sequences. This information plays a crucial role in current approaches to language processing tasks like question answering and instruction generation. We describe how to leverage language models for non-linguistic perception and control tasks. Our approach casts labeling and decision-making as inference in probabilistic graphical models in which language models parameterize prior distributions over labels, decisions and parameters, making it possible to integrate uncertain observations and incomplete background knowledge in a principled way. Applied to semantic segmentation, household navigation, and activity recognition tasks, this approach improves predictions on rare, out-of-distribution, and structurally novel inputs.
Introduction
Large Language Models (LLMs) like GPT-4 are renowned for their ability to generate human-like text, but their utility extends far beyond linguistic tasks. Researchers from MIT CSAIL have demonstrated that the "world knowledge" embedded in LLMs can serve as a powerful common-sense engine for non-linguistic tasks, such as helping a robot navigate a home or improving computer vision systems. Their framework, LaMPP (Language Models as Probabilistic Priors), treats LLMs as probabilistic priors within structured graphical models, providing a principled way to integrate human-like intuition into perception and action systems.
The Challenge: Data Scarcity and Distributional Shift
Traditional machine learning models are typically narrow and data-hungry. A vision model trained on a dataset where bathrooms are underrepresented might fail to recognize a shower curtain because it lacks the contextual understanding that shower curtains and bathtubs frequently co-occur. While humans use general world knowledge to fill these gaps, AI models often break down when encountering "out-of-distribution" (OOD) data—scenarios that differ from their specific training sets. LaMPP addresses this by leveraging the vast textual knowledge of LLMs to provide the necessary context that visual or robotic training data might lack.
The LaMPP Framework
Instead of simply "chaining" models—where a vision model produces a text description for an LLM to process—LaMPP uses the LLM to parameterize a prior distribution within a Bayesian graphical model. This allows the system to maintain a rigorous mathematical framework while benefiting from the LLM's common sense.
The approach follows a four-step process:
- Base Model Selection: A standard task-specific model (e.g., a visual classifier) provides a likelihood $p(x | y)$.
- Label Space Design: Relationships between variables are defined, often including latent variables like "room type" to capture global context.
- LM Querying: The LLM is prompted to score the plausibility of different configurations (e.g., the likelihood of a "nightstand" being in a "bedroom"). These scores are converted into a probability distribution $p(y)$.
- Inference: The LLM's prior is combined with the base model's observations using Bayes' rule: $p(y | x) \propto p(y) p(x | y)$.
Experimental Results
LaMPP was evaluated across three distinct domains: semantic image segmentation (SUN RGB-D), household navigation (Habitat Challenge), and video action recognition (CrossTask). The results consistently showed that adding an LLM prior boosts performance, particularly in zero-shot and out-of-distribution settings.
- Semantic Segmentation: LaMPP improved the detection of rare classes like "shower curtains" by 18.9 IoU points. In OOD settings, it helped identify "nightstands" (+8.92 IoU) even when they were missing from the training data.
- Navigation: The framework achieved a 66.5% success rate in finding objects, significantly outperforming the base model's 52.7%.
- Action Recognition: For video tasks, LaMPP improved the accuracy of identifying held-out action transitions by an average of 8.2%.
Discussion and Future Work
By treating LLMs as common-sense prior engines, LaMPP effectively corrects for systematic biases in training datasets. However, challenges remain, including the sensitivity of LLMs to prompt phrasing and the need for better calibration of their probability scores. Future research will focus on automating prompt engineering and extending these priors to more complex, continuous control tasks in robotics.
Summary updated