Action Representation for Spatial Generalization in Robot World Models

Yuhang SunarXiv 2026 · arXiv

How should a robot represent its actions so that what it learns in one corner of a table applies to the other? Or, more challenging still, so that what it learns about peeling an apple applies to peeling a pear?

In a new paper titled "Action Representation for Spatial Generalization in Robot World Models," Yuhang Sun from the University of Washington explores how the "frame of reference" we choose for robot actions dictates how well a model can generalize across space and physical configurations.

TL;DR

This study demonstrates that spatial generalization in robots isn't just about better scene reconstruction; it's about choosing action coordinates that preserve physical relationships. While local, surface-relative representations significantly improve learning efficiency and decision-making, they create a "predictive sufficiency" trap: if you simplify coordinates too much and discard essential physical details (like material thickness), you create an information-theoretic ceiling that no amount of data can overcome.

What Problem Does This Paper Address?

Robot "world models" try to predict the consequences of actions. However, a major hurdle is spatial generalization. If a robot learns a "cutting law" in world coordinates, that knowledge is tied to a specific spot in the room. If the object moves, the world-coordinate description changes, even if the physical interaction is identical.

Researchers often use "canonicalization"—standardizing inputs into a local object frame—to solve this. But this paper identifies a critical gap: we don't fully understand the trade-offs. If we make a representation too invariant (ignoring everything but the local patch), we might throw away the very information needed to predict the outcome (like the thickness of the peel or the curvature of the fruit).

Methods

The author developed a controlled robotic peeling benchmark using a geometric material-removal simulator. This allowed for precise manipulation of variables that are often "muddied" in real-world robotics, such as exact contact geometry and material depth.

The study compared three primary action representations:

  1. World-frame (W): Actions defined in fixed workspace coordinates.
  2. Object-frame (B): Actions defined relative to the fruit's center.
  3. Surface-relative (S): Actions defined relative to the specific local patch being contacted.

The predictive model used was an action-conditioned PointNet, which processed surface points (positions, normals, and thickness) alongside the blade's path. Crucially, the author didn't just look at prediction error (MSE); they evaluated "Decision Regret"—how much worse the robot's chosen action was compared to the theoretical optimum.

Datasets

The study utilized procedurally generated datasets:

  • Fruit Geometries: 14 base geometries created using directional polynomial modes to simulate irregular, "organic" shapes.
  • Training/Test Sets: Typically 600–1000 training interactions and 280–300 test interactions per experiment.
  • Paired Interventions: A specialized set of 90 pairs where only peel thickness was changed to test "predictive sufficiency."

Key Insights and Ideas

  • Invariance vs. Sufficiency: The paper distinguishes between two requirements. Consequence-preserving invariance means the model shouldn't care if you move the whole fruit-blade system (Figure 1a). Predictive sufficiency means the model must care if the local state changes, like peel thickness (Figure 1b). Invariance vs. Sufficiency
Figure 1. Consequence-preserving and consequence-changing variations in object-centric manipulation. (a) Two global poses of the same fruit–blade interaction produce equivalent material-removal consequences in fruit- attached coordinates. (b) Changing the local peel thickness under an identical blade motion and removal-depth field changes the relationship between residual peel and exposed flesh. The two cases illustrate the complementary requirements of invariance to consequence-irrelevant spatial transformations and retention of physical information necessary for consequence prediction.
  • The "Input Collision" Dead End: If a representation discards thickness, two physically different interactions (one that clears the peel and one that doesn't) look identical to the model. This "collision" creates an irreducible error that more data or a bigger neural network cannot fix.
  • The Asymmetry of Noise: Not all coordinate errors are equal. The model is far more sensitive to errors along the surface normal (depth) than tangential shifts. This suggests that "robustness" isn't a single number; it depends on the direction of the uncertainty.
  • Accuracy $\neq$ Decision Quality: A model might have a higher average error but still be better at ranking actions. Surface-relative models were much better at picking the "best" action even when their absolute predictions were slightly off.

Key Results

  • Learning Efficiency: Object-relative coordinates (B) maintained a constant error regardless of where the fruit was placed, while World-frame (W) error spiked by ~16.7% as the range of possible poses increased.
  • Superior Generalization: On irregular fruits, the Surface-relative (S) model achieved an NRMSE of 0.56, a 36.6% reduction in error compared to the Object-relative model (0.89).
  • Decision Regret: In counterfactual ranking tests, the Surface-relative representation reduced "normalized regret" by 65.3% compared to the World-frame baseline (Figure 6). Decision Regret Comparison
  • The Cost of "Blindness": When peel thickness was withheld, error increased by 54.8%, proving that coordinate normalization is useless if it discards the underlying physics.
Decision Regret Comparison

Limitations and Future Work

The primary limitation is the use of a geometric simulator. While it captures the spatial complexity of peeling, it does not model the complex "fracture mechanics," adhesion, or force transients of real-world cutting.

The author suggests that the next logical steps are:

  1. Closing the Loop: Moving from short-horizon "strokes" to long-horizon, closed-loop control.
  2. Real-world Sensing: Testing these principles with actual tactile and force feedback data.
  3. Richer Mechanics: Incorporating more realistic material interactions beyond simple geometric removal.

By focusing on why certain representations work rather than just which one is best, this paper provides a principled foundation for designing robots that can truly "understand" the physical consequences of their actions across any environment.

Summary updated

Abstract

Spatial generalization in robot world models depends on whether action representations capture the physical relationships that remain valid across changing spatial configurations. Yet different coordinates can describe the same interaction, whereas similar motions can produce different outcomes when local physical conditions change. How these distinctions affect learned prediction remains insufficiently understood. This study evaluates matched action representations defined in world, object, and surface coordinate frames using a controlled robotic peeling benchmark. Global pose and contact relocation experiments reveal a cost associated with limited training data when equivalent interactions receive different spatial descriptions. Removing information about peel thickness creates an irreducible ambiguity because identical predictive inputs can correspond to different material removal outcomes. Coordinates defined relative to the surface improve consequence prediction and counterfactual action selection under accurate local observations, but learned predictors remain vulnerable to uncertainty in the coordinate frame. Joint centering of the surface and tool removes sensitivity to translation of their common origin, whereas incorrect contact geometry and actual execution deviations introduce different failure mechanisms. These results show that effective spatial generalization depends on preserving the relationships between actions and states that determine physical consequences, rather than simply minimizing coordinate variation. This perspective provides a principled basis for designing robot world models that reuse physical knowledge across spatial configurations while retaining the distinctions essential to prediction and planning.
BibTeX
@misc{sun2026action,
  author = {Yuhang Sun},
  title = {{Action Representation for Spatial Generalization in Robot World Models}},
  year = {2026},
  eprint = {2611.11869},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11869},
}
Figure