LaMPP: Language Models as Probabilistic Priors for Perception and Action

Belinda Z. Li, William Chen, Pratyusha Sharma, Jacob AndreasSubmission to ICLR 2024 · Rejected

TL;DR

Instead of prompting Large Language Models (LLMs) to make direct decisions in non-linguistic perception and robotics tasks, LAMPP (Language Models as Probabilistic Priors) extracts structured prior distributions from LMs and injects them into classical probabilistic graphical models. This principled combination of top-down common-sense priors with bottom-up sensory uncertainty drastically boosts out-of-distribution and rare-class performance while cutting LLM API inference costs by orders of magnitude.

What Problem Does This Paper Address?

Perception and robotic control systems often fail when deployed in the wild on rare, ambiguous, or out-of-distribution (OOD) inputs—such as confusing a shower curtain for a window curtain, or searching for a TV in a bathroom. While large text corpora contain vast repositories of common-sense world knowledge (e.g., plates belong in kitchens, shower curtains go near bathtubs), bridging text-based LLMs to perceptual models is difficult.

Current "model-chaining" techniques (like Socratic Models) convert perceptual outputs into natural language and ask the LLM to predict the answer directly. However, this strategy discards graded perceptual uncertainty, struggles with formal reasoning over continuous features, and incurs massive inference costs by querying the LLM for every single decision.

Methods

LAMPP reinterprets LLMs as sources of prior distributions within Probabilistic Graphical Models (PGMs), combining "top-down" linguistic common sense with "bottom-up" perceptual models via Bayes’ rule:

  1. Base Model Selection: Any standard observation model can be used (e.g., RedNet/CLIPSeg for segmentation, STUBBORN for navigation, or an HMM for action recognition).
  2. Label Space & Graphical Model Design: Introduce latent variables that compactly explain correlations. For example, in image segmentation, object labels are coupled by a latent room type $r$ and perceptual confusion variables $d$.
  3. Querying the LM Offline: Rather than prompting per test instance, the LLM is queried ahead of time to populate conditional probability tables (e.g., $p(\text{object} \mid \text{room})$ or Dirichlet hyperparameters $\alpha$ over action transitions $y \to y'$). Queries evaluate the relative probability of tokens like plausible vs. implausible.
  4. Bayesian Inference: Decisions are made by running standard probabilistic inference over the resulting graph (e.g., max-marginal inference or MAP parameter estimation), naturally balancing perceptual evidence against prior plausibility.
       [ Latent Context / Room (r) ]
             |               \
             v                v
   [ True Object (y₁) ]     [ True Object (y₂) ]
             |                        |
             v                        v
  [ Perceived Label (d₁) ]  [ Perceived Label (d₂) ]
             |                        |
             v                        v
   [ Image Segment (x₁) ]   [ Image Segment (x₂) ]

Datasets

The authors evaluate LAMPP across three distinct modalities and tasks:

  • SUN RGB-D: Indoor RGB-D images used for semantic segmentation, tested under in-distribution (ID), out-of-distribution (removing co-occurring bed–nightstand pairs during training), and zero-shot (ZS) transfer settings.
  • Habitat Challenge ObjectNav (modified): A simulated 3D indoor navigation benchmark. The authors augmented evaluation episodes by manually annotating high-level room labels.
  • CrossTask: Instructional videos covering diverse procedural tasks (e.g., make pancakes, change a tire) used for action recognition and temporal segmentation.

Key Insights and Ideas

  • Query-Once Efficiency: Socratic Models query the LLM repeatedly per inference step or frame. LAMPP queries the LM once per domain/vocabulary to construct prior tables. This dropped segmentation query costs from $3,112.00 to $1.86 (over 99.9% cost reduction).
  • Graded Uncertainty Trumps Textual Logic: In navigation, ablating the prior probability $p(\text{goal} \mid \text{room})$ during the object verification phase degraded performance below the baseline. Combining probabilistic perceptual scores with graded LM priors is mathematically and empirically superior to passing discrete text strings back and forth.
  • Selective Intervention on Rare/Biased Data: In semantic segmentation, base models frequently failed on underrepresented categories. LAMPP left accurate, frequent predictions untouched while dramatically correcting rare confusions using scene context (e.g., knowing an object in a bathroom is a shower curtain, not a window curtain).

Key Results

Task Base Model Setup Baseline Metric + LAMPP Cost Comparison
Semantic Segmentation (mIoU) RedNet In-Distribution (ID) 47.8 mIoU 48.3 mIoU (+18.9 max class $\Delta$) **$1.86** vs. $3112.00 (SM)
RedNet Out-of-Distribution (OOD) 33.8 mIoU 34.0 mIoU (+8.92 max class $\Delta$) $1.86
CLIPSeg Zero-Shot (ZS) 27.7 mIoU 28.2 mIoU (+13.3 max class $\Delta$) $1.86
Object Navigation (Success Rate) Uniform Prior Zero-Shot (ZS) 52.1% (Class-Avg) 66.5% (+14.4%) **$0.03** vs. $0.54 (SM)
Action Segmentation (Step Recall) HSMM Zero-Shot (ZS) 44.4% (Class-Avg) 45.7% (+1.3% class, +1.9% freq) $1.09

Note: Socratic Models (SM) on RedNet ID degraded mIoU to 37.5 due to uncalibrated text hallucinations.

Limitations and Future Work

  • Task Formalization Constraints: LAMPP requires domains that can be formalized as graphical models where state variables map meaningfully to natural language strings.
  • Domain Mismatch: Highly specialized fields (such as fine-grained medical diagnostics or low-level robotic motor torque control) lack commonsense representations in general text pre-training corpora.
  • LM Prompt and Ordering Biases: In sequential action tasks, GPT-3 demonstrated a strong recency and presentation-order bias (preferring whichever order actions were listed in the prompt), limiting the efficacy of action-transition priors without substantial prompt tuning.

Summary updated

Abstract

Language models trained on large text corpora encode rich distributional information about real-world environments and action sequences. This information plays a crucial role in current approaches to language processing tasks like question answering and instruction generation. We describe how to leverage language models for *non-linguistic* perception and control tasks. Our approach casts labeling and decision-making as inference in probabilistic graphical models in which language models parameterize prior distributions over labels, decisions and parameters, making it possible to integrate uncertain observations and incomplete background knowledge in a principled way. Applied to semantic segmentation, household navigation, and activity recognition tasks, this approach improves predictions on rare, out-of-distribution, and structurally novel inputs.
BibTeX
@misc{li2023lampp,
  author = {Belinda Z. Li and William Chen and Pratyusha Sharma and Jacob Andreas},
  title = {{LaMPP: Language Models as Probabilistic Priors for Perception and Action}},
  year = {2023},
  eprint = {2302.02801},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.02801},
}

Loading related content...

Figure