Subtle predictive movements reveal actions regardless of social context

Emalie McMahon, Charles Zheng, Francisco Pereira, Ray González, Leslie G. Ungerleider, Maryam Vaziri-PashkamJournal of Vision 2019 · DOI

TL;DR

Humans can predict others' actions remarkably early, even before movement begins, using subtle cues distributed across their entire body. This predictive ability remains consistent across cooperative and competitive social contexts, suggesting that our intentions are revealed regardless of whether we aim to collaborate or compete.

What Problem Does This Paper Address?

We're all pretty good at predicting what someone is about to do, whether it's a barista handing us coffee or a soccer player about to kick the ball. This ability is crucial for smooth social interactions. But how do we do it? What parts of the body give away our intentions, and does the social situation—like whether we're cooperating or competing—change the information we reveal? Previous research has shown we can predict actions, but the precise spatiotemporal profile of these predictive cues and how social context modulates them remained unclear. This paper aims to precisely map out when and where in the body predictive information becomes available, and whether this changes when the social goal shifts from cooperation to competition.

Methods

The researchers employed a two-pronged approach:

  1. Psychophysics Experiment: Human participants watched short video clips of individuals performing a reaching task. These clips were cut at various points before the actual finger lift-off, and participants had to predict the target direction. This allowed them to measure human predictive accuracy over time.
  2. Machine Learning Classification: A linear Support Vector Machine (SVM) was trained on optical flow data extracted from the video frames. This classifier aimed to predict the movement direction based on local motion information. This allowed for a more precise, pixel-level analysis of where predictive information was located.

A novel searchlight analysis technique, borrowed from neuroimaging, was also used. This involved training and testing a classifier within small, localized regions (searchlights) of the video frame to map the spatiotemporal distribution of predictive information across the initiator's body.

Datasets

The study used videos collected from participants performing a reaching game. These videos served as the primary dataset for both the human psychophysics experiment and the machine learning analyses. No new public benchmarks were introduced.

Key Insights and Ideas

  • Predictive Power of Early Movement: Even before a person's finger lifts off the table, there's enough subtle movement information for both humans and a machine learning classifier to predict their intended action. This highlights how much our intentions are "written" on our bodies before we fully commit to an action.
  • Distributed Information: Predictive cues aren't confined to a single body part like the hand or arm. Instead, information is spread across the entire body, from the head to the torso and limbs.
  • Social Context Doesn't Hide Intentions: Surprisingly, the social context (cooperation vs. competition) didn't significantly alter where or when predictive information became available. This suggests that even when we might want to conceal our intentions in competition, subtle movements still betray them.
  • Speed vs. Information Availability: While competitive actions were faster, cooperative actions were slower. This difference in speed, rather than the social context itself, seemed to explain some of the differences in how early predictions could be made. Cooperative initiators might slow down to give their partners more time to process the information.
  • Machine Learning as a Proxy for Human Perception: The performance of the linear SVM classifier closely mirrored human performance, suggesting that simple optical flow information might be a key component of how humans predict actions. This correlation implies that the visual information humans use is tightly linked to local motion patterns.

Key Results

  • Human Prediction: Participants could predict the target direction with significantly above-chance accuracy at time points up to 150 ms before the finger lift-off in the cooperative condition, and around 83 ms before lift-off in the competitive condition.
  • SVM Prediction: The SVM classifier achieved above-chance accuracy at approximately 150 ms before movement start in the cooperative condition and 83 ms before lift-off in the competitive condition.
  • T75 (Time to 75% Accuracy): The time at which both humans and the SVM reached 75% accuracy was comparable. While the SVM showed an earlier T75 in the cooperative condition, this difference was largely accounted for by the slower movement speed in cooperation.
  • Searchlight Analysis: Information was found to be widely distributed across the initiator's body in both conditions. While some initial differences were observed before FDR correction, these largely disappeared after correction, indicating minimal differences in the distribution of information between cooperation and competition.

Limitations and Future Work

  • Deception and Free Choice: The study's participants were instructed rather than freely choosing their actions, and deception was not a factor. Future research could explore how these elements, common in real-world social interactions, might modulate predictive cues.
  • Machine Learning Model Simplicity: The study used a linear classifier. While this choice was made for interpretability, more complex models (like deep neural networks) might capture more nuanced human predictive abilities.
  • Timescale of Information Integration: The classifier was trained on single frames of optical flow. Humans might integrate information over multiple frames, and future work could investigate this temporal integration more directly.
  • Generalizability: The findings are based on a specific reaching task. Further research is needed to determine if these results generalize to other types of actions and social contexts.
  • Interpreting Searchlight Maps: While searchlight analysis reveals where information is available, it doesn't definitively tell us how humans are using that information. Future work could use these maps to constrain hypotheses about human predictive strategies.

Summary updated

Abstract

Humans have a remarkable ability to predict the actions of others. To address what information enables this prediction and how the information is modulated by social context, we used videos collected during an interactive reaching game. Two participants (an "initiator" and a "responder") sat on either side of a plexiglass screen on which two targets were affixed. The initiator was directed to tap one of the two targets, and the responder had to either beat the initiator to the target (competition) or arrive at the same time (cooperation). In a psychophysics experiment, new observers predicted the direction of the initiators' reach from brief clips, which were clipped relative to when the initiator began reaching. A machine learning classifier performed the same task. Both humans and the classifier were able to determine the direction of movement before the finger lift-off in both social conditions. Further, using an information mapping technique, the relevant information was found to be distributed throughout the body of the initiator in both social conditions. Our results indicate that we reveal our intentions during cooperation, in which communicating the future course of actions is beneficial, and also during competition despite the social motivation to reveal less information.
BibTeX
@article{mcmahon2019subtle,
  author = {Emalie McMahon and Charles Zheng and Francisco Pereira and Ray González and Leslie G. Ungerleider and Maryam Vaziri-Pashkam},
  title = {{Subtle predictive movements reveal actions regardless of social context}},
  journal = {Journal of Vision},
  year = {2019},
  doi = {10.1167/19.7.16},
}
Figure