Object Centric Local Coordinate Frames for Invariant Learning

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 121 papers28 conceptsfrozen 2026-10-04

In standard computer vision and robotics, we typically represent the world using a single, global coordinate system. While this is mathematically convenient, it forces neural networks to learn the same concepts repeatedly: a cup at the center of a room looks different to a model than a cup in the corner, and a robot pushing a box north must learn a different set of motor commands than one pushing it east.

A growing body of research suggests that the key to data-efficient and generalizable intelligence lies in object-centric local coordinate frames. Instead of representing the entire scene in one grid, these models assign each object its own private coordinate system. By shifting the perspective from "where is this in the world?" to "what does this look like relative to itself?", researchers are enabling models to achieve natural invariance to translation, rotation, and scale.

The Problem of Global Entanglement

When a model uses global coordinates, the "what" (identity) and the "where" (pose) are entangled. As Hong-Xing Yu and Jiajun Wu point out in their work on uOCF, this entanglement forces a model to learn the appearance of an object at every possible location in a scene. This is highly inefficient.

Early connectionist models, such as those proposed by Maximilian Riesenhuber and Peter Dayan in Neural Models for Part-Whole Hierarchies, addressed this by delegating spatial transformations to an external attentional system. By separating the "attentional eye-position" from the representation of the object itself, the high-level units could focus exclusively on identity. This separation of concerns ensures that recognition remains invariant to where an object sits in the visual field.

Disentangling Identity from Pose

The modern implementation of this idea involves transforming global data into Local Canonical Coordinates (LCC). In 3D object detection, Yang You and colleagues demonstrated in Canonical Voting that regressing these rotation-invariant local coordinates is far more robust than trying to guess a bounding box's global orientation directly.

This concept is taken further in the Invariant Slot Attention (ISA) framework by Ondrej Biza and Thomas Kipf. ISA assigns each "slot" (a latent representation of an object) its own reference frame. By transforming global coordinates into relative grids based on the object's estimated pose, the model's decoder can learn a single appearance template that applies to every instance of a category, regardless of its position or scale. This "slot-centric" reasoning allows the model to generalize to new scene layouts without needing to see the same object in every possible configuration.

Invariance in Dynamics and Interaction

The benefits of local frames extend beyond static recognition into the realm of movement and physics. In robotics, Dana H. Ballard pioneered the use of task frames—coordinate systems attached directly to the object being manipulated. By specifying actions in the object's own frame, complex global trajectories become simple, invariant commands.

This principle is echoed in recent work on human-object interaction. Nilesh Kulkarni and Leonidas Guibas introduced NIFTY, which uses neural interaction fields defined relative to objects. Similarly, Liang An and Hongwen Zhang developed ManiDext, which represents hand-object manipulation within the object's local coordinate system. In both cases, the model doesn't have to learn how to "sit on a chair" at every coordinate in the room; it learns the relationship between the human and the chair's local frame, which remains constant no matter where the chair is moved.

For multi-agent systems, Miltiadis Kofinas and Efstratios Gavves showed in LoCS that transforming data into roto-translated local frames grants networks Galilean invariance. This allows models to predict the trajectories of colliding particles or traffic participants by focusing on relative interactions rather than arbitrary global positioning.

Why It Works: The Intuition of Canonicalization

The fundamental reason this approach works is canonicalization. By shifting to a local frame, the model effectively "undoes" the transformations of the environment.

Imagine trying to learn the physics of a bouncing ball. In global coordinates, the ball's path is a complex curve that changes depending on where you stand. In the ball's own frame, the physics of its rotation and internal deformation become much simpler to model. As Jonathan Scholz and David Wingate argue in their work on Physics-Based Reinforcement Learning, modeling dynamics within an object's local frame enables immediate generalization across different spatial orientations.

By decoupling the "intrinsic" properties of an object (its shape, appearance, and physical behavior) from its "extrinsic" properties (its position, orientation, and scale), these models mirror the way humans perceive the world: as a collection of persistent entities that maintain their identity even as they move through space.

Papers in this version

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_object_centric_local_coordinate_frames_f_a1ad9078,
  title = {Object Centric Local Coordinate Frames for Invariant Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/object-centric-local-coordinate-frames-for-invariant-learning/rdr_a1ad907850ee429fa804e63d612bc353},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_object_centric_local_coordinate_frames_f_a1ad9078_v1,
  title = {Object Centric Local Coordinate Frames for Invariant Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/object-centric-local-coordinate-frames-for-invariant-learning/rdr_a1ad907850ee429fa804e63d612bc353/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}