Distinguishing Self, Other, and Autonomy From Visual Feedback: A Combined Correlation and Acceleration Transfer Analysis

Berkay Demirel, Clément Moulin-Frier, Xerxes D. Arsiwalla, Paul F. M. J. Verschure, Martí Sánchez-FiblaFrontiers in Human Neuroscience 2021 · DOI

Abstract

In cognitive science, Theory of Mind (ToM) is the mental faculty of assessing intentions and beliefs of others and requires, in part, to distinguish incoming sensorimotor (SM) signals and, accordingly, attribute these to either the self-model, the model of the other, or one pertaining to the external world, including inanimate objects. To gain an understanding of this mechanism, we perform a computational analysis of SM interactions in a dual-arm robotic setup. Our main contribution is that, under the common fate principle, a correlation analysis of the velocities of visual pivots is shown to be sufficient to characterize "the self" (including proximo-distal arm-joint dependencies) and to assess motor to sensory influences, and "the other" by computing clusters in the correlation dependency graph. A correlational analysis, however, is not sufficient to assess the non-symmetric/directed dependencies required to infer autonomy, the ability of entities to move by themselves. We subsequently validate 3 measures that can potentially quantify a metric for autonomy: Granger causality (GC), transfer entropy (TE), as well as a novel "Acceleration Transfer" (AT) measure, which is an instantaneous measure that computes the estimated instantaneous transfer of acceleration between visual features, from which one can compute a directed SM graph. Subsequently, autonomy is characterized by the sink nodes in this directed graph. This study results show that although TE can capture the directional dependencies, a rectified subtraction operation denoted, in this study, as AT is both sufficient and computationally cheaper.

Introduction

Before an agent can engage in complex social interactions or develop a "Theory of Mind" (ToM), it must solve a fundamental perceptual problem: distinguishing between its own body, other agents, and passive objects in the environment. This distinction is the bedrock of agency. While traditional approaches often rely on internal motor copies (forward models) or data-heavy deep learning, this research explores the minimal computational requirements to detect autonomy—the ability of an entity to move by itself—using only visual feedback.

Methodology and Experimental Setup

The study utilizes a simulated "Sally-Anne" robotics environment featuring two 3-DOF robotic arms and a sliding ball. Visual input is simplified into Sensory Interest Points (SIPs), which are discrete pivots representing joints and object centers. This abstraction allows the researchers to focus on the dynamics of motion rather than high-dimensional image processing.

The simulated dual-arm robotic environment (A) and the temporal evolution of velocity and haptic signals (B).

The researchers evaluated three primary statistical frameworks to analyze these SIPs:

  1. Correlational Analysis: Using Pearson’s correlation to identify "what moves together," adhering to the Gestalt principle of common fate. This effectively maps the kinematic structure of the "self" and identifies clusters representing "others."
  2. Causality Measures: Standard tools like Granger Causality (GC) and Transfer Entropy (TE) were tested to identify directed, non-symmetric relationships.
  3. Acceleration Transfer (AT): A novel, instantaneous measure proposed by the authors. AT is based on the physical principle that a passive object cannot initiate its own acceleration; it can only receive it. By calculating the "rectified subtraction" of accelerations between points, the system builds a directed Sensorimotor Graph.

Key Insights: From Structure to Agency

A critical finding of the study is that correlation is sufficient for structure but insufficient for agency. Because correlation is symmetric, it can identify that a hand and a ball are moving in tandem, but it cannot determine which entity is the source of the motion. To identify autonomy, one must look for directed influences.

Correlation matrix (A) revealing kinematic structures and directed influence maps (B-E) illustrating the transfer of motion between entities.

The researchers introduced the "Sink Node" Principle: in a directed graph of acceleration transfer, a passive object (like the ball) acts as a sink. It receives acceleration from autonomous agents but never initiates a transfer to another entity. While Transfer Entropy can detect these directions, it is computationally expensive and requires significant historical data. In contrast, the AT measure leverages Newtonian physics ($F=ma$) to provide a computationally efficient, instantaneous alternative.

Results and Performance

The experimental results demonstrated that the AT measure significantly outperforms traditional causality measures in autonomy detection. In complex interaction scenarios involving random movements, AT maintained high F1 scores (between 0.77 and 0.91), whereas Granger Causality performance plummeted to as low as 0.16. Furthermore, correlation analysis successfully characterized the proximo-distal dependencies of the robotic arms, effectively allowing the agent to "recognize" its own body structure without prior kinematic models.

Conclusion and Future Work

This work provides a "minimalist" framework for agency detection in robotics. By using Acceleration Transfer, robots can efficiently annotate their sensory experience to distinguish between self-initiated actions and external influences.

Future research will focus on scaling this model from discrete SIPs to raw, high-dimensional video pixels and testing the robustness of AT in 3D environments with occlusions. This developmental approach could eventually allow robots to learn "who did what to whom" in social contexts, serving as a precursor to more advanced social learning and cooperation.

Summary updated

Abstract

In cognitive science, Theory of Mind (ToM) is the mental faculty of assessing intentions and beliefs of others and requires, in part, to distinguish incoming sensorimotor (SM) signals and, accordingly, attribute these to either the self-model, the model of the other, or one pertaining to the external world, including inanimate objects. To gain an understanding of this mechanism, we perform a computational analysis of SM interactions in a dual-arm robotic setup. Our main contribution is that, under the common fate principle, a correlation analysis of the velocities of visual pivots is shown to be sufficient to characterize "the self" (including proximo-distal arm-joint dependencies) and to assess motor to sensory influences, and "the other" by computing clusters in the correlation dependency graph. A correlational analysis, however, is not sufficient to assess the non-symmetric/directed dependencies required to infer autonomy, the ability of entities to move by themselves. We subsequently validate 3 measures that can potentially quantify a metric for autonomy: Granger causality (GC), transfer entropy (TE), as well as a novel "Acceleration Transfer" (AT) measure, which is an instantaneous measure that computes the estimated instantaneous transfer of acceleration between visual features, from which one can compute a directed SM graph. Subsequently, autonomy is characterized by the sink nodes in this directed graph. This study results show that although TE can capture the directional dependencies, a rectified subtraction operation denoted, in this study, as AT is both sufficient and computationally cheaper.
BibTeX
@inproceedings{demirel2021distinguishing,
  author = {Berkay Demirel and Clément Moulin-Frier and Xerxes D. Arsiwalla and Paul F. M. J. Verschure and Martí Sánchez-Fibla},
  title = {{Distinguishing Self, Other, and Autonomy From Visual Feedback: A Combined Correlation and Acceleration Transfer Analysis}},
  booktitle = {Frontiers in Human Neuroscience},
  year = {2021},
  doi = {10.3389/fnhum.2021.560657},
}
Figure