TL;DR
This paper introduces ORCA (Object-Relative Coordinates fixed to Anchor), a simple yet powerful shift in how reinforcement learning agents "see" objects—moving from absolute global coordinates to relative frames centered on a task-relevant anchor (like the player). By pairing this representation with a transformer-based architecture (ORBiT), the authors achieve human-level performance on Atari and dominant results in complex continuous control tasks like Foosball, while significantly improving robustness to spatial shifts.
What Problem Does This Paper Address?
In Object-Centric Reinforcement Learning (OCRL), agents receive state information as a list of objects with properties like position and velocity. Traditionally, these positions are given in absolute coordinates (e.g., "the player is at (10, 20), the goal is at (50, 20)").
The problem is that an agent’s optimal decision almost always depends on relative distances, not absolute ones. If a player needs to move 5 pixels right to catch a ball, that rule should hold whether they are at the top-left or bottom-right of the screen. With absolute coordinates, the agent must effectively relearn the same "catch" strategy for every possible screen location. This leads to sample inefficiency and extreme fragility: if you shift the entire game screen by a few pixels, an agent trained on absolute coordinates often suffers a catastrophic collapse in performance.
Methods
The authors propose two main components:
- ORCA (Object-Relative Coordinates fixed to Anchor): Instead of feeding the agent raw coordinates, they transform the input into a frame of reference centered on a designated anchor object (typically the player or a critical entity like the ball). This involves subtracting the anchor's position and velocity from all other objects and, in environments with orientation, applying a rotation matrix to align the frame with the anchor's heading.
- ORBiT (Object-Relative inductive-Bias Transformer): To process these relative inputs, they use a transformer encoder. Transformers are a natural fit for OCRL because they are order-invariant (they don't care about the sequence of objects) and count-flexible (they can handle a varying number of objects in a scene).
The architecture (shown in Figure 3) takes the ORCA-transformed object set, embeds them into a high-dimensional space, processes them through transformer layers, and finally aggregates them (via mean-pooling) to feed policy and value heads.
Datasets
The researchers evaluated their approach across three distinct domains:
- Atari 2600 (ALE): 13 games were used for training. To test robustness, they used the HackAtari benchmark, which introduces perturbations like shifting the game frame or changing background logic.
- Foosball: A high-dimensional (8-DoF) continuous control environment where the agent must coordinate multiple rods to play against an opponent.
- Tag: A controlled 2D environment designed to test how the choice of anchor and the number of distractors affect performance.
Key Insights and Ideas
- Spatial Inductive Bias: Humans don't think in coordinates; we think in relations ("the keys are next to the bowl"). ORCA bakes this human-like spatial reasoning into the model as an inductive bias.
- Invariance by Design: By using relative coordinates, the agent becomes mathematically invariant to global translations and rotations. This means it doesn't just "learn" to be robust to shifts; it is robust by construction.
- The Transformer Advantage: While relative coordinates help standard MLPs, the gain is much larger for Transformers. The authors suggest this is because Transformers can attend to each object's relative features independently, whereas MLPs struggle with the rigid, concatenated vector format.
- Anchor Choice Matters: In the "Tag" environment, the authors showed that the anchor must be "decision-relevant." Anchoring to the player or the target led to high performance, while anchoring to a random moving distractor caused the agent to fail.
Key Results
- Atari Performance: ORBiT achieved an aggregate Human-Normalized Score (HNS) Interquartile Mean (IQM) of 1.10, surpassing the pixel-based PPO baseline (0.69) and the absolute-coordinate transformer (0.54).
- Robustness: When the game screen was translated (shifted), absolute-coordinate models lost 30-50% of their performance. ORBiT remained essentially unaffected, losing only about 1-4%.
- Foosball Dominance: In head-to-head matches, ORBiT scored 49,700 goals against a standard MLP baseline, which managed only 330 goals. This demonstrates that the approach scales to complex, high-speed continuous control.
Limitations and Future Work
- 2D Constraint: The current implementation is focused on 2D environments. Extending this to 3D spaces is a necessary step for real-world robotics.
- Manual Anchoring: Currently, the "anchor" object is selected manually based on domain knowledge. The authors suggest that learning to identify the most relevant anchor automatically is a key area for future research.
- Perfect Perception: The study assumes a "perfect" object extractor. Future work should investigate how ORBiT handles the noise and "flickering" typical of real-world computer vision systems.
Summary updated