Flow Snapshot Neurons in Action: Deep Neural Networks Generalize to Biological Motion Perception

Shuangpeng Han, Ziyu Wang, Mengmi Zhang

TL;DR

Researchers have developed the Motion Perceiver (MP), a novel neural network architecture that mimics the human ability to recognize actions from minimal "point-light" displays. By training on natural RGB videos and focusing purely on patch-level optical flow, MP achieves a massive 29% accuracy improvement over existing AI models and demonstrates behavioral alignment with human perception.

What Problem Does This Paper Address?

Humans possess a remarkable ability called Biological Motion Perception (BMP). We can identify a person walking, running, or jumping even if we only see a few moving dots representing their joints—a "point-light display." Crucially, we do this with zero specific training on these minimal stimuli.

Current AI models, however, are notoriously brittle. While they excel at recognizing actions in high-resolution RGB videos, they fail to generalize when visual information is stripped away. Most models over-rely on static "appearance" cues (e.g., recognizing a soccer player because of the green grass and white ball) rather than the underlying motion dynamics. The authors set out to bridge this gap by creating a model that learns robust, invariant motion representations that generalize from natural videos to the sparse stimuli used in neuroscience.

Figure 1: Humans excel at biological motion perception (BMP) tasks with zero training, while current AI models struggle with poor generalization performance. AI models are trained to recognize actions from natural RGB videos and tested using BMP stimuli on point-light displays, which come in two forms: Joint videos, which display only the detected joints of actors in white dots, and Sequential position actor videos (SP), where light points in white are randomly positioned between joints and reallocated to other random positions on the limb in subsequent frames (Sec. 3.1). Note that skeletons, shown in gray in the example video, are not visible to humans or AI models during testing. The generalization performance of both humans and models is assessed after varying five properties in temporal and visual dimensions. See Appendix, Sec. A.1 for example videos.

Methods

The Motion Perceiver (MP) architecture moves away from standard pixel-level analysis, instead utilizing patch-level optical flows extracted from a frozen DINO-pretrained Vision Transformer (ViT). The model processes these flows through two specialized neural pathways:

  1. Flow Snapshot Neurons: Inspired by neuroscience, these neurons use a Slot Attention mechanism. They compete to "bind" to specific, prototypical motion patterns (e.g., the specific curve of an arm swing). This allows the model to store and recognize consistent motion "snapshots" across different video patches.
  2. Motion Invariant Neurons: These neurons aggregate motion magnitudes across four directions (+x, -x, +y, -y) over time. This makes the representation robust to variations in action speed and minor temporal distortions.

The outputs from these pathways are fused across multiple temporal scales to capture both rapid movements and long-term dependencies.

Figure 2: Architecture of our proposed Motion Perceiver (MP) model. Given a reference patch (yellow or green example patches), MP computes its patch-level optical flow (red arrows, Sec. 2.1) on the feature maps extracted from DINO [10]. Subsequently, these flows are processed through flow snapshot neurons (Sec. 2.2) and motion invariant neurons (Sec. 2.3) in two pathways. Activations from both groups of neurons are then integrated for action classification (Sec. 2.4). Time embeddings (T Emb.) are used in the feature fusion process.

Datasets

The authors introduced a massive new benchmark for this field:

  • BMP Dataset: Contains 62,656 video stimuli across 24 conditions, manipulating factors like temporal order, resolution, and the number of visible points. It includes "Joint videos" (dots on joints) and "Sequential Position (SP) videos" (dots randomly placed on limbs to eliminate local motion signals).
  • Human Psychophysics Data: The authors collected recognition data from 88 human subjects via Amazon Mechanical Turk to serve as a gold-standard "upper bound" for generalization.
  • Standard Benchmarks: The model was also evaluated on point-light versions of NTU RGB+D 60 and NW-UCLA to test its performance on standard computer vision tasks.

Key Insights and Ideas

  • Appearance is a Crutch: By forcing the model to rely only on optical flow, the researchers prevented it from "cheating" using background or texture cues, leading to much better generalization to sparse dot displays.
  • Human-Like Error Patterns: Unlike standard 3D CNNs, MP's performance degrades in ways that mirror human behavior. For example, both humans and MP are significantly more affected by reversing the temporal order of a "Joint" video than an RGB video.
  • Robustness to Sparsity: The "Enhanced MP" (En-MP) version, which integrates features from multiple layers of the DINO backbone, maintains high accuracy even when the number of visible points is reduced to just five, closely matching human robustness.
Figure 4: Our model demonstrates human-like robustness under reduced visual information. Top-1 action recognition accuracy is a function of the number of points (P) in Joint (J) videos. Results from RGB test videos are at the leftmost. The colored shaded region represents the standard error across all action classes.

Key Results

  • Massive Generalization Leap: MP outperformed all existing AI baselines (like SlowFast, I3D, and MViT) by up to 29% in top-1 accuracy on the BMP conditions.
  • Benchmark Superiority: On the NTU RGB+D 60 (Joint) dataset, MP achieved 20.38% accuracy compared to the next best baseline's 6.89%. On NW-UCLA (Joint), it hit 42.83% vs. the baseline's 16.59%.
  • Temporal Sensitivity: Figure 3 highlights that while temporal shuffling impairs both humans and MP, MP retains a significantly higher "above-chance" performance than other models, suggesting it captures the essence of the motion rather than just frame-to-frame correlations.
Figure 3: Temporal orders and resolutions matter in generalization performance on RGB and Joint videos. Stimuli encompass RGB and Joint (J) videos. Short forms include R (reversal), S (shuffle), F (frames), and P (points) in Sec. 3.1. Error bars indicate the standard error of the top-1 accuracy across different action classes.

Limitations and Future Work

While MP is a major step forward, the authors note it primarily models the dorsal pathway (motion perception) and lacks the ventral pathway (form perception) integration found in the human brain. The model also does not currently account for top-down influences like attention or prior expectations.

Future work could involve expanding the BMP dataset to include more complex social interactions and testing the model's robustness in highly noisy real-world environments where visual information is severely limited (e.g., extreme low-light surveillance).

Summary updated

Abstract

Biological motion perception (BMP) refers to humans' ability to perceive and recognize the actions of living beings solely from their motion patterns, sometimes as minimal as those depicted on point-light displays. While humans excel at these tasks without any prior training, current AI models struggle with poor generalization performance. To close this research gap, we propose the Motion Perceiver (MP). MP solely relies on patch-level optical flows from video clips as inputs. During training, it learns prototypical flow snapshots through a competitive binding mechanism and integrates invariant motion representations to predict action labels for the given video. During inference, we evaluate the generalization ability of all AI models and humans on 62,656 video stimuli spanning 24 BMP conditions using point-light displays in neuroscience. Remarkably, MP outperforms all existing AI models with a maximum improvement of 29% in top-1 action recognition accuracy on these conditions. Moreover, we benchmark all AI models in point-light displays of two standard video datasets in computer vision. MP also demonstrates superior performance in these cases. More interestingly, via psychophysics experiments, we found that MP recognizes biological movements in a way that aligns with human behaviors. Our data and code are available at https://github.com/ZhangLab-DeepNeuroCogLab/MotionPerceiver.
BibTeX
@inproceedings{han2024flow,
  author = {Shuangpeng Han and Ziyu Wang and Mengmi Zhang},
  title = {{Flow Snapshot Neurons in Action: Deep Neural Networks Generalize to Biological Motion Perception}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=btuHzsAVsK},
}
Figure