Predicate Hierarchies Improve Few-Shot State Classification

Emily Jin, Joy Hsu, Jiajun Wu

Abstract

Robotic state classification—the ability to interpret spatial relations and object states from visual data—is often hampered by the combinatorial explosion of object-predicate pairs. We present PHIER (Predicate HIERarchies), a framework that improves few-shot generalization by embedding image-predicate pairs into a hyperbolic latent space. By leveraging semantic hierarchies derived from large language models, PHIER achieves significant performance gains in out-of-distribution tasks and real-world transfer.

Introduction

In robotic environments, understanding the state of objects (e.g., "is the drawer open?") and their spatial relations (e.g., "is the cup to the right of the plate?") is fundamental for task execution. However, traditional supervised models struggle with the vast variety of possible object-predicate combinations. Collecting exhaustive training data is infeasible, and existing vision-language models often lack the precision required for fine-grained spatial reasoning. PHIER addresses these challenges by treating predicates not as independent labels, but as structured hierarchies.

Methodology: The PHIER Framework

PHIER introduces a novel approach to state classification through three key components:

  1. Object-Centric Encoding: Utilizing a MaskCLIP-based approach, the model focuses on relevant entities by masking the rest of the image. This ensures that the neural network's attention is directed solely at the objects involved in the query.
  2. LLM-Guided Semantic Priors: PHIER uses Large Language Models (LLMs) to extract semantic relationships between predicates. This knowledge is used to supervise the vision model, teaching it which predicates are similar and which are more specific (e.g., "OnRight" as a subset of "NextTo").
  3. Hyperbolic Latent Space: The core innovation lies in the use of a Poincaré ball for embedding. Hyperbolic geometry is naturally suited for representing hierarchical structures, allowing general predicates to reside near the center while specific "child" predicates are positioned toward the perimeter.
The PHIER framework: training on known states, inferring a predicate hierarchy, and generalizing to novel predicates.

Key Insights: Geometry and Language as Priors

The effectiveness of PHIER stems from its architectural priors. By forcing the model to operate within a hyperbolic space, the concept of hierarchy is baked into the mathematical representation. This allows the model to infer the meaning of novel predicates based on their proximity to known ones in the latent space.

Detailed view of the inferred predicate hierarchy, showing the relationship between learned and novel spatial concepts.

Furthermore, the use of LLMs as a scaffold demonstrates that explicit manual labeling of hierarchies is unnecessary. The model successfully extracts pre-existing linguistic knowledge to guide visual learning, effectively disentangling object identification from relational classification.

Evaluation and Results

PHIER was evaluated across simulated environments (CALVIN and BEHAVIOR) and real-world benchmarks (BEHAVIOR Vision Suite).

  • Out-of-Distribution (OOD) Performance: PHIER outperformed state-of-the-art methods by 22.5 percentage points on the CALVIN dataset and 8.3 points on BEHAVIOR.
  • Sim-to-Real Transfer: In zero-shot settings on real-world images, PHIER surpassed larger models like ViperGPT and BLIP-2, demonstrating robust generalization from simulation.
  • Few-Shot Efficiency: With only five examples of a novel predicate, PHIER achieved performance comparable to GPT-4V, despite being a much smaller, specialized model.

Conclusion and Future Work

PHIER demonstrates that incorporating semantic hierarchies into robotic vision models significantly enhances their ability to generalize to new tasks and environments. Future research will explore deriving hierarchies directly from visual data and extending the framework to dynamic scenes involving temporal predicates.

Summary updated

Abstract

State classification of objects and their relations is core to many long-horizon tasks, particularly in robot planning and manipulation. However, the combinatorial explosion of possible object-predicate combinations, coupled with the need to adapt to novel real-world environments, makes it a desideratum for state classification models to generalize to novel queries with few examples. To this end, we propose PHIER, which leverages predicate hierarchies to generalize effectively in few-shot scenarios. PHIER uses an object-centric scene encoder, self-supervised losses that infer semantic relations between predicates, and a hyperbolic distance metric that captures hierarchical structure; it learns a structured latent space of image-predicate pairs that guides reasoning over state classification queries. We evaluate PHIER in the CALVIN and BEHAVIOR robotic environments and show that PHIER significantly outperforms existing methods in few-shot, out-of-distribution state classification, and demonstrates strong zero- and few-shot generalization from simulated to real-world tasks. Our results demonstrate that leveraging predicate hierarchies improves performance on state classification tasks with limited data.
BibTeX
@inproceedings{jin2025predicate,
  author = {Emily Jin and Joy Hsu and Jiajun Wu},
  title = {{Predicate Hierarchies Improve Few-Shot State Classification}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=lxu8Vz6cLs},
}
Figure