Abstract
Robotic state classification—the ability to interpret spatial relations and object states from visual data—is often hampered by the combinatorial explosion of object-predicate pairs. We present PHIER (Predicate HIERarchies), a framework that improves few-shot generalization by embedding image-predicate pairs into a hyperbolic latent space. By leveraging semantic hierarchies derived from large language models, PHIER achieves significant performance gains in out-of-distribution tasks and real-world transfer.
Introduction
In robotic environments, understanding the state of objects (e.g., "is the drawer open?") and their spatial relations (e.g., "is the cup to the right of the plate?") is fundamental for task execution. However, traditional supervised models struggle with the vast variety of possible object-predicate combinations. Collecting exhaustive training data is infeasible, and existing vision-language models often lack the precision required for fine-grained spatial reasoning. PHIER addresses these challenges by treating predicates not as independent labels, but as structured hierarchies.
Methodology: The PHIER Framework
PHIER introduces a novel approach to state classification through three key components:
- Object-Centric Encoding: Utilizing a MaskCLIP-based approach, the model focuses on relevant entities by masking the rest of the image. This ensures that the neural network's attention is directed solely at the objects involved in the query.
- LLM-Guided Semantic Priors: PHIER uses Large Language Models (LLMs) to extract semantic relationships between predicates. This knowledge is used to supervise the vision model, teaching it which predicates are similar and which are more specific (e.g., "OnRight" as a subset of "NextTo").
- Hyperbolic Latent Space: The core innovation lies in the use of a Poincaré ball for embedding. Hyperbolic geometry is naturally suited for representing hierarchical structures, allowing general predicates to reside near the center while specific "child" predicates are positioned toward the perimeter.
Key Insights: Geometry and Language as Priors
The effectiveness of PHIER stems from its architectural priors. By forcing the model to operate within a hyperbolic space, the concept of hierarchy is baked into the mathematical representation. This allows the model to infer the meaning of novel predicates based on their proximity to known ones in the latent space.
Furthermore, the use of LLMs as a scaffold demonstrates that explicit manual labeling of hierarchies is unnecessary. The model successfully extracts pre-existing linguistic knowledge to guide visual learning, effectively disentangling object identification from relational classification.
Evaluation and Results
PHIER was evaluated across simulated environments (CALVIN and BEHAVIOR) and real-world benchmarks (BEHAVIOR Vision Suite).
- Out-of-Distribution (OOD) Performance: PHIER outperformed state-of-the-art methods by 22.5 percentage points on the CALVIN dataset and 8.3 points on BEHAVIOR.
- Sim-to-Real Transfer: In zero-shot settings on real-world images, PHIER surpassed larger models like ViperGPT and BLIP-2, demonstrating robust generalization from simulation.
- Few-Shot Efficiency: With only five examples of a novel predicate, PHIER achieved performance comparable to GPT-4V, despite being a much smaller, specialized model.
Conclusion and Future Work
PHIER demonstrates that incorporating semantic hierarchies into robotic vision models significantly enhances their ability to generalize to new tasks and environments. Future research will explore deriving hierarchies directly from visual data and extending the framework to dynamic scenes involving temporal predicates.
Summary updated