Reframing Object-Centric Reinforcement Learning: From Absolute to Relative Representations

Cedric Derstroff, Jannis Blüml, Janosch Moos, Michael Girstl, Zihan Ye, Debora Clever, Kristian KerstingarXiv 2026 · arXiv

TL;DR

This paper introduces ORCA (Object-Relative Coordinates fixed to Anchor), a simple yet powerful shift in how reinforcement learning agents "see" objects—moving from absolute global coordinates to relative frames centered on a task-relevant anchor (like the player). By pairing this representation with a transformer-based architecture (ORBiT), the authors achieve human-level performance on Atari and dominant results in complex continuous control tasks like Foosball, while significantly improving robustness to spatial shifts.

What Problem Does This Paper Address?

In Object-Centric Reinforcement Learning (OCRL), agents receive state information as a list of objects with properties like position and velocity. Traditionally, these positions are given in absolute coordinates (e.g., "the player is at (10, 20), the goal is at (50, 20)").

The problem is that an agent’s optimal decision almost always depends on relative distances, not absolute ones. If a player needs to move 5 pixels right to catch a ball, that rule should hold whether they are at the top-left or bottom-right of the screen. With absolute coordinates, the agent must effectively relearn the same "catch" strategy for every possible screen location. This leads to sample inefficiency and extreme fragility: if you shift the entire game screen by a few pixels, an agent trained on absolute coordinates often suffers a catastrophic collapse in performance.

Figure 1: With ORCA inputs, the agent only needs to learn strategies for similar situations once, enabling direct generalization. When the player (Asterix ) is in the same row as the target collectible (cauldron ), e.g., the cauldron (1, 5) is right of Asterix (1, 0), the absolute coordinates of the objects should not influence the agent’s decision. The agent should behave the same even if the cauldron is at (3, 9), which is also on the right of Asterix (3, 4), but on a different platform. Object-Relative Coordinates fixed to Anchor (ORCA) performs this transformation in object space, simplifying the agent’s decision.

Methods

The authors propose two main components:

  1. ORCA (Object-Relative Coordinates fixed to Anchor): Instead of feeding the agent raw coordinates, they transform the input into a frame of reference centered on a designated anchor object (typically the player or a critical entity like the ball). This involves subtracting the anchor's position and velocity from all other objects and, in environments with orientation, applying a rotation matrix to align the frame with the anchor's heading.
  2. ORBiT (Object-Relative inductive-Bias Transformer): To process these relative inputs, they use a transformer encoder. Transformers are a natural fit for OCRL because they are order-invariant (they don't care about the sequence of objects) and count-flexible (they can handle a varying number of objects in a scene).

The architecture (shown in Figure 3) takes the ORCA-transformed object set, embeds them into a high-dimensional space, processes them through transformer layers, and finally aggregates them (via mean-pooling) to feed policy and value heads.

Figure 3: The ORBiT architecture, showing the coordinate transform followed by a transformer encoder and policy/critic heads.

Datasets

The researchers evaluated their approach across three distinct domains:

  • Atari 2600 (ALE): 13 games were used for training. To test robustness, they used the HackAtari benchmark, which introduces perturbations like shifting the game frame or changing background logic.
  • Foosball: A high-dimensional (8-DoF) continuous control environment where the agent must coordinate multiple rods to play against an opponent.
  • Tag: A controlled 2D environment designed to test how the choice of anchor and the number of distractors affect performance.

Key Insights and Ideas

  • Spatial Inductive Bias: Humans don't think in coordinates; we think in relations ("the keys are next to the bowl"). ORCA bakes this human-like spatial reasoning into the model as an inductive bias.
  • Invariance by Design: By using relative coordinates, the agent becomes mathematically invariant to global translations and rotations. This means it doesn't just "learn" to be robust to shifts; it is robust by construction.
  • The Transformer Advantage: While relative coordinates help standard MLPs, the gain is much larger for Transformers. The authors suggest this is because Transformers can attend to each object's relative features independently, whereas MLPs struggle with the rigid, concatenated vector format.
  • Anchor Choice Matters: In the "Tag" environment, the authors showed that the anchor must be "decision-relevant." Anchoring to the player or the target led to high performance, while anchoring to a random moving distractor caused the agent to fail.

Key Results

  • Atari Performance: ORBiT achieved an aggregate Human-Normalized Score (HNS) Interquartile Mean (IQM) of 1.10, surpassing the pixel-based PPO baseline (0.69) and the absolute-coordinate transformer (0.54).
  • Robustness: When the game screen was translated (shifted), absolute-coordinate models lost 30-50% of their performance. ORBiT remained essentially unaffected, losing only about 1-4%.
  • Foosball Dominance: In head-to-head matches, ORBiT scored 49,700 goals against a standard MLP baseline, which managed only 330 goals. This demonstrates that the approach scales to complex, high-speed continuous control.
Figure 4: Aggregated Atari results showing ORCA significantly boosting performance for both MLP and Transformer architectures.

Limitations and Future Work

  • 2D Constraint: The current implementation is focused on 2D environments. Extending this to 3D spaces is a necessary step for real-world robotics.
  • Manual Anchoring: Currently, the "anchor" object is selected manually based on domain knowledge. The authors suggest that learning to identify the most relevant anchor automatically is a key area for future research.
  • Perfect Perception: The study assumes a "perfect" object extractor. Future work should investigate how ORBiT handles the noise and "flickering" typical of real-world computer vision systems.

Summary updated

Abstract

Object-centric reinforcement learning agents typically receive each object's position in a fixed global frame, even though their decisions usually depend on where objects are relative to one another. These absolute views often fail to capture the local context that is most relevant to the agent's behavior. As a result, an agent must relearn the same response at every location where a configuration can occur, and its performance collapses when the scene is merely shifted. In this work, we propose a shift toward Object-Relative Coordinates fixed to Anchor (ORCA), where entities are described in a relative frame of reference. To operationalize this idea, we introduce Object-Relative inductive-Bias Transformers (ORBiT), a transformer-based architecture for processing anchor-relative inputs in reinforcement learning. Our results suggest these relative representations improve performance for both MLP and transformer agents. Furthermore, we demonstrate that ORBiT outperforms the pixel-based PPO baseline, achieving an aggregate HNS IQM greater than 1.0 over 13 Atari games. ORBiT learns the game of Foosball from scratch, dominating all baselines. In our controlled Tag environment, we further show that ORCA extends to rotations, that ORBiT generalizes to unseen numbers of distractors, and that these gains require a decision-relevant anchor.
BibTeX
@misc{derstroff2026reframing,
  author = {Cedric Derstroff and Jannis Blüml and Janosch Moos and Michael Girstl and Zihan Ye and Debora Clever and Kristian Kersting},
  title = {{Reframing Object-Centric Reinforcement Learning: From Absolute to Relative Representations}},
  year = {2026},
  eprint = {2610.28197},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.28197},
}
Figure