Abstract
Artificial agents often struggle to distinguish between sensory changes caused by their own actions (reafference) and those caused by external environmental factors (exafference). This paper introduces a formal causal framework that utilizes the "null-action" as a counterfactual baseline to disentangle these effects. By learning a forward model that predicts the outcome of doing nothing, agents can isolate their own causal influence from ambient world dynamics. We evaluate this approach across physics-based and visual environments, demonstrating that agents can effectively "discover" their own bodies and separate self-generated signals from external influences.
Introduction
In robotics and artificial intelligence, agents are inundated with sensory data. A fundamental challenge is determining the source of sensory change: if a robot moves its arm and observes a shift in its camera feed, was that change a direct result of its motor command, or did an external object move simultaneously? Biological organisms solve this problem through mechanisms like the "efference copy," allowing them to distinguish between self-generated and externally-generated stimuli.
Standard Reinforcement Learning (RL) agents typically rely on associative models that fail to make this causal distinction. This lack of disentanglement hinders robustness—for instance, a model trained on flat ground may fail on a slope because it hasn't separated the effect of gravity from its own movements. Furthermore, a true "sense of agency" and the ability to perform higher-level functions like planning or imitation require an explicit understanding of what the agent itself is responsible for.
Methods
The proposed framework leverages Pearl’s do-calculus to formally define reafference. The core innovation is the use of the null-action ($∅$)—the act of "doing nothing" (e.g., a noop or zero torque)—as a counterfactual baseline.
The Reafferent Forward Model
The agent maintains a forward model $f_\theta$ that predicts the next state given the current state and an action. By querying this model with the null-action, the agent estimates the exafferent effect: $\hat{\delta}_\u2205 = f_\theta(x, \u2205)$. The reafferent effect is then defined as the additional change caused by a specific action $a$ relative to this baseline: $\hat{\delta}_a = f_\theta(x, a) - \hat{\delta}_\u2205$.
The model is trained using Stochastic Gradient Descent (SGD) to minimize the Mean Squared Error (MSE) between the predicted total effect ($\hat{\delta}_a + \hat{\delta}_\u2205$) and the actual observed change $\delta = x' - x$. This allows the agent to learn the distinction purely from experience without a pre-defined body map.
Experiments and Results
The framework was evaluated in three distinct simulated environments:
- Cartpole: The agent successfully separated the constant exafferent force of gravity from the reafferent impulses applied to the cart.
- Atari Freeway: In this high-dimensional visual task, the model naturally isolated the "self" (the chicken) from the "environment" (moving cars). The reafferent map highlighted only the chicken pixels, effectively "discovering" the agent's body.
- Artificial Ape: This 3D environment simulated complex perspective shifts. The agent could distinguish its own rotation from the rotation of external platforms, even when they occurred simultaneously.
Discussion
This work frames reafference as a "Difference in Differences" problem, where the agent's influence is the marginal change over a "do-nothing" counterfactual. A key insight is that body maps can emerge as a byproduct of causal disentanglement; the agent identifies its "body" as the set of features it can causally influence.
While promising, the current approach is limited to single-step predictions. Future work is needed to handle multi-step reafference, where actions have delayed effects. Additionally, defining a "null-action" in continuous or highly complex biological systems may require more sophisticated modeling. Nonetheless, by formalizing the act of doing nothing, this research provides a mathematically grounded path toward agents with a robust sense of agency.
Summary updated