Resolving Spatial Reference Frame Ambiguity

48 concepts36 papers1 proposal

When a human tells a robot to "pick up the mug on the left," the instruction is fundamentally incomplete. Left from whose perspective? The human's? The robot's? Or perhaps the intrinsic left of a nearby object, like a computer monitor?

This ambiguity stems from the fact that spatial language does not map directly to absolute physical coordinates. Instead, as early computational linguistics work by Patrick Olivier et al. in A Computational View of the Cognitive Semantics of Spatial Prepostions established, interpreting spatial prepositions requires resolving distinct spatial reference frames. These frames are typically categorized into three types:

  1. Intrinsic (Object-centric): Defined by the landmark object's own internal axes (e.g., "in front of the car," facing its headlights).
  2. Relative (Viewer-centric / Deictic): Defined by the viewpoint of an observer (e.g., "to my left").
  3. Extrinsic (World-centric): Defined by external environmental anchors or global coordinates (e.g., "north of the building").

When an AI agent or robot fails to distinguish between these frames, it suffers from a viewpoint misalignment or "perspective gap." As Yuqi Bu et al. note in Walk in Others' Shoes with a Single Glance: Human-Centric Visual Grounding with Top-View Perspective Transformation, this gap becomes acute when humans and robots face each other or stand at varying angles, turning simple directional commands into a complex coordinate-transformation puzzle.

Why Standard Vision-Language Models Fail

Modern Vision-Language Models (VLMs) excel at identifying objects, but they struggle significantly with spatial relationships. Benchmarks like Visual Spatial Reasoning, introduced by Fangyu Liu et al., reveal that while humans easily navigate shifting perspectives, state-of-the-art models often plateau.

Standard models fail because they treat spatial relations as simple binary truths based on absolute 2D image coordinates. They lack an internal mechanism to represent how space transforms when a viewpoint rotates. As Michael Ogezi argues in SpaRE: Enhancing Spatial Reasoning in Vision-Language Models with Synthetic Data, without explicit grounding in a specific coordinate system, models cannot resolve conflicting but equally valid spatial interpretations. Furthermore, the COMFORT benchmark by Zheyuan Zhang et al. in Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference under Ambiguities demonstrates that models are highly brittle and struggle to adopt non-egocentric perspectives, often defaulting to a rigid viewer-centric bias.

Bridging the Gap via Geometric Perspective-Taking

To solve this, researchers have turned to explicit geometric transformations that mimic human cognitive perspective-taking. If the robot can mentally "walk in the speaker's shoes," the coordinate misalignment vanishes.

One intuitive mechanism is view rotation. The REP (Reasoning from your Perspective) framework by Cheng Shi 0001 et al. in Spatial and Visual Perspective-Taking via View Rotation and Relation Reasoning for Embodied Reference Understanding combines 3D depth estimation with multimodal reasoning to mentally rotate the agent's perspective until it aligns with the human sender's viewpoint. By transforming the visual scene into the speaker's coordinate system, relative terms like "left" or "right" can be computed directly.

Another approach projects the 3D scene into a canonical, viewpoint-invariant space. In Walk in Others' Shoes with a Single Glance, the authors developed a method that maps egocentric views into an abstract top-view perspective transformation. By looking at the scene from a bird's-eye perspective, the spatial relationships between the human, the robot, and the objects become explicitly clear, eliminating the angular confusion inherent in front-facing cameras.

[Human Viewpoint]  ──┐
                     ├─► [Canonical Top-View Projection] ─► Unambiguous Spatial Grounding
[Robot Viewpoint]  ──┘

Learning Perspective-Invariant Features

In robotic imitation learning, the perspective gap manifests as the "correspondence problem." A robot watching a human demonstration from a third-person camera angle must map those visual coordinates to its own internal motor commands.

Instead of explicitly rotating the geometry, we can force the underlying neural network to ignore the viewpoint altogether. Bradly C. Stadie et al. pioneered this using domain confusion in Third Person Imitation Learning. By training a discriminator to guess which camera angle a feature came from, and optimizing the policy to confuse this discriminator, the network is forced to extract features that are entirely invariant to perspective.

Similarly, Xinlei Pan et al. utilized a Feature Disentanglement Network (FDN) in Zero-shot Imitation Learning from Demonstrations for Legged Robot Visual Navigation. By separating "what" is happening in the task from "where" the camera is positioned, the robot can execute zero-shot imitation learning across mismatched visual frames.

Multi-Frame Reasoning and Cognitive Selection

Rather than forcing a single perspective, the most robust systems model multiple reference frames simultaneously and resolve the ambiguity dynamically.

This multi-frame approach has deep roots. In Automatic Depiction of Spatial Descriptions, Patrick Olivier et al. built a system that represented objects with explicit internal axes to generate multiple valid interpretations (intrinsic vs. deictic) of ambiguous descriptions. Later, Reinhard Moratz proposed in Intuitive linguistic Joint Object Reference in Human-Robot Interaction evaluating potential targets across both intrinsic and relative frames simultaneously, calculating the likelihood of each target under both systems to resolve linguistic ambiguity.

In modern architectures, Chan Hee Song formalized this with RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics, a framework that explicitly calculates spatial relationships across ego-centric, world-centric, and object-centric perspectives at the same time. By evaluating the scene through all three lenses, the model can decode which frame makes the most sense given the context of the instruction.

Finally, because human preferences shift depending on the environment, Mingjiang Liu et al. introduced a perspective-corrected generation model in Perspective-Corrected Spatial Referring Expression Generation for Human-Robot Interaction. Instead of rigidly adhering to one viewpoint, it dynamically selects a reference frame based on predicted human resolution patterns and landmark characteristics, doubling communication success by aligning the robot's language with human cognitive expectations.

Key Foundational Work

A Computational View of the Cognitive Semantics of Spatial Prepostions (1994) — contributed 2 concept elements to this direction.
Generating Coordinated Natural Language and 3D Animations for Complex Spatial Explanations (1998) — contributed 2 concept elements to this direction.
Automatic Depiction of Spatial Descriptions (1994) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_spatial_reference_frame_ambiguity_3d601640,
  title = {spatial reference frame ambiguity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spatial-reference-frame-ambiguity/rdr_3d6016401ba6435d80a74e98bdf99d4b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_spatial_reference_frame_ambiguity_3d601640_v1,
  title = {spatial reference frame ambiguity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/spatial-reference-frame-ambiguity/rdr_3d6016401ba6435d80a74e98bdf99d4b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake