Moving Forward by Moving Backward: Embedding Action Impact over Action Semantics

Kuo-Hao Zeng, Luca Weihs, Roozbeh Mottaghi, Ali Farhadi

Most embodied AI agents fail when their "legs" break or the floor gets slippery because they rely on fixed definitions of what actions do. This paper introduces the Action Adaptive Policy (AAP), which allows agents to learn the actual physical impact of their actions on-the-fly, enabling them to navigate successfully even when actions are disabled or radically altered at test time.

The Flaw in the Action-Stability Assumption

In the world of Embodied AI, researchers typically operate under the Action-Stability (AS) assumption: the idea that an action like "MoveAhead" will always move the agent forward by a fixed distance (e.g., 0.25 meters).

However, the real world is messy. A robot might encounter a slippery wet floor, a carpet that creates more friction than hardwood, or a mechanical failure like a broken wheel. Standard reinforcement learning agents are brittle in these scenarios. If the "MoveAhead" action suddenly starts rotating the agent instead of moving it forward, the agent's performance collapses because it cannot adapt its internal model of what that action "means."

The Action Adaptive Policy (AAP)

To address this, the authors propose shifting the focus from action semantics (the label of the action) to action impact (what the action actually does in the environment).

Expected vs. actual action impacts and the AAP architecture.

The AAP architecture consists of three main modules:

  1. State Encoder: A standard module that processes visual observations and task goals into a hidden state.
  2. Action-Impact Encoder: Instead of assuming it knows what an action does, the agent observes the visual state change ($o_{t-1} \to o_t$) after executing an action. A recurrent neural network updates a latent embedding for that specific action based on the observed outcome. Crucially, the agent starts an episode knowing nothing about its actions and must "learn by doing."
  3. Order-Invariant (OI) Head: Standard linear policy heads are biased because they expect actions in a specific order (the "Linear Layer Trap"). To fix this, AAP uses a Transformer-based head without positional encodings. This treats the set of action embeddings as an unordered collection, allowing the agent to pick the action with the most useful impact regardless of its original label.

Experimental Validation

The researchers evaluated AAP across several challenging environments, including AI2-THOR (RoboTHOR) for Point Navigation and Object Navigation, the high-fidelity Habitat simulator, and PettingZoo for simplified 2D continuous force dynamics.

Key Insights and Results

  • Labels are Liars: In a "drift" scenario, the label "MoveAhead" might actually result in a 30-degree right turn. By ignoring the labels and focusing on latent embeddings of state-changes, the agent can realize that a different action (e.g., formerly "MoveBackward") is now the best way to move toward the goal.
  • Exploitation over Compensation: Unlike "robust" policies that learn conservative, slow strategies to handle noise, AAP aims to fully exploit the available actions. If a "MoveAhead" action suddenly moves the agent twice as far, AAP learns to reach the goal in half the time.
  • Extreme Drifts and Disabled Actions: In PointNav, when movement and rotation magnitudes were shifted far outside the training range, AAP maintained a Success Rate (SR) near 100%, while baselines dropped below 30%. When tested in a "broken robot" scenario where certain rotation actions were disabled, AAP achieved a 96-98% SR, whereas the best baseline (Meta-RL) struggled around 29%.
  • Real-World Transfer: In RoboTHOR real-world scenes with unseen physical drifts, AAP achieved a 65% SR, doubling the performance of the best-tuned baseline (30%).

Limitations and Future Work

Because the agent must learn action impacts on-the-fly, it faces a "Cold Start" problem. It necessarily performs worse in the first few steps of an episode compared to an agent that assumes perfect action stability, as it has to "probe" the environment to understand its new body. Furthermore, while AAP handles rotation drifts perfectly, its performance begins to degrade if movement magnitudes become extremely large, as the visual changes become too erratic for the encoder.

Future directions include extending this approach to continuous action spaces and more complex manipulation tasks where the impact of an action might depend heavily on the specific object being handled.

Summary updated

Abstract

A common assumption when training embodied agents is that the impact of taking an action is stable; for instance, executing the ``move ahead'' action will always move the agent forward by a fixed distance, perhaps with some small amount of actuator-induced noise. This assumption is limiting; an agent may encounter settings that dramatically alter the impact of actions: a move ahead action on a wet floor may send the agent twice as far as it expects and using the same action with a broken wheel might transform the expected translation into a rotation. Instead of relying that the impact of an action stably reflects its pre-defined semantic meaning, we propose to model the impact of actions on-the-fly using latent embeddings. By combining these latent action embeddings with a novel, transformer-based, policy head, we design an Action Adaptive Policy (AAP). We evaluate our AAP on two challenging visual navigation tasks in the AI2-THOR and Habitat environments and show that our AAP is highly performant even when faced, at inference-time, with missing actions and, previously unseen, perturbed action spaces. Moreover, we observe significant improvement in robustness against these actions when evaluating in real-world scenarios.
BibTeX
@inproceedings{zeng2023moving,
  author = {Kuo-Hao Zeng and Luca Weihs and Roozbeh Mottaghi and Ali Farhadi},
  title = {{Moving Forward by Moving Backward: Embedding Action Impact over Action Semantics}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=vmjctNUSWI},
}
Figure