LINGO-Space: Language-Conditioned Incremental Grounding for Space

Dohyun Kim, Nayoung Oh, Deokmin Hwang, Daehyung ParkAAAI 2024 · DOI

Introduction: The Challenge of Space Grounding

In robotics, identifying a specific object (instance grounding) is a well-established task. However, space grounding—localizing a specific coordinate based on spatial relations—presents significant hurdles. Human instructions like "to the left of the bench and front of the pillar" are often ambiguous in distance, referentially complex (e.g., multiple benches), and compositionally dense.

LINGO-Space in action on a Boston Dynamics Spot robot, localizing a target based on multi-part spatial instructions.

LINGO-Space addresses these challenges by combining Large Language Models (LLMs) with a probabilistic spatial model. It treats space grounding as an incremental process, narrowing down target locations by resolving overlapping constraints.

Methodology

The LINGO-Space architecture is divided into three primary modules that translate visual and linguistic inputs into a precise spatial probability map.

The LINGO-Space architecture, featuring the Scene-Graph Generator, Semantic Parser, and Spatial-distribution Estimator.
  1. Scene-Graph Generator: Visual inputs are converted into a structured graph. The system utilizes Grounding DINO for object detection and CLIP to extract features for nodes (objects) and edges (spatial relationships).
  2. LLM-based Semantic Parser: ChatGPT is employed to decompose natural language instructions into structured relationship tuples. This allows the system to handle linguistic variety by translating "messy" human speech into a standardized geometric language.
  3. Spatial-distribution Estimator: This module predicts a probabilistic distribution rather than a single point. It uses polar distributions (Gaussian for distance, Von Mises for angle) to model human-like spatial descriptions. A Graph Power Sequence (GPS) layer—combining Message Passing Neural Networks and Transformers—updates the probability map incrementally as each instruction phrase is processed.

Key Insights

  • Polar vs. Cartesian Coordinates: Representing relations via angle and distance is more effective than X-Y coordinates. It naturally captures "directional uncertainty," where a direction is known but distance is not.
  • Incremental Bayesian Refinement: Each phrase in a complex instruction (e.g., "...and near the lamp") acts as a filter, reshaping the existing probability distribution to "zoom in" on the target.
  • LLMs as Structural Parsers: Instead of relying on LLMs for geometric reasoning, LINGO-Space uses them to handle linguistic complexity, feeding structured data into specialized neural networks for high-precision localization.

Experimental Results

The framework was evaluated across several benchmarks, including CLIPort, PARAGON, SREM, and a new LINGO-Space dataset designed to test "close" and "far" predicates.

  • Benchmark Performance: LINGO-Space achieved state-of-the-art results in 11 out of 12 standard tasks.
  • Compositional Complexity: In tasks involving up to six sequential spatial phrases, LINGO-Space maintained a success score of 90.5, vastly outperforming the next best baseline (28.5).
  • Real-World Validation: The system was successfully deployed on a Boston Dynamics Spot robot, demonstrating its ability to navigate real environments based on natural language commands.

Limitations and Future Directions

Currently, the model focuses on linguistic accuracy but does not fully account for physical volumes. A predicted target might be mathematically correct but physically occupied by an object. Future iterations aim to integrate 3D volumetric reasoning and adapt to dynamic scenes where objects may move during the instruction process.

Summary updated

BibTeX
@inproceedings{kim2024lingo,
  author = {Dohyun Kim and Nayoung Oh and Deokmin Hwang and Daehyung Park},
  title = {{LINGO-Space: Language-Conditioned Incremental Grounding for Space}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i9.28898},
}
Figure