Motion-Scenario Decoupling for Rat-Aware Video Position Prediction: Strategy and Benchmark

Xiaofeng Liu, Jiaxin Gao, Yaohua Liu, Nenggan Zheng, Risheng Liu

Introduction

In the field of computer vision, predicting human actions—such as swinging a golf club or performing household chores—has reached a high level of maturity. However, predicting the movements of biological organisms, particularly "bio-robots" (small animals like rats controlled via Brain-Computer Interfaces), presents a significantly more complex challenge. Unlike humans, whose actions often follow predictable, rational patterns, animal behavior is characterized by high levels of uncertainty and is influenced by both biological variability and environmental constraints.

This paper addresses the challenge of animal movement prediction by introducing RatPose, a specialized benchmark dataset, and the Dual-stream Motion-Scenario Decoupling (DMSD) framework. By explicitly separating environmental context from movement patterns, the proposed method achieves a substantial improvement in prediction accuracy over existing state-of-the-art models.

The Challenge of Animal Position Prediction

Predicting the future position of a bio-robotic rat is essentially a "hard mode" version of action recognition. Existing research, such as the LMBRD dataset, focuses primarily on identifying current behaviors (e.g., grooming vs. walking). For practical applications in search-and-rescue or military operations, however, the critical requirement is position prediction: determining where the animal will be in the next several seconds.

Two primary factors complicate this task:

  1. Individual Differences: Biological diversity means that different rats—or even the same rat at different times—exhibit varying speeds and behaviors based on hunger, fatigue, or temperament.
  2. Environmental Scenarios: A rat's movement is constrained and directed by its surroundings. A maze imposes different movement logic than an open field.

Standard deep learning models often conflate these two factors, leading to poor generalization when either the subject or the environment changes.

The RatPose Benchmark

To facilitate research in this area, the authors developed RatPose, the first high-quality dataset specifically designed for rat-aware video position prediction. The dataset includes:

  • Diverse Scenarios: Five distinct environments, including lab simulations, mazes, open fields, and challenging conditions such as light and dense fog.
  • Biological Diversity: Data from six individual rats to account for behavioral variance.
  • Labeled Sequences: 1,023 data pairs where video sequences are mapped to five future position categories: Top, Down, Left, Right, and Middle.

The DMSD Framework

The core contribution of this work is the Dual-stream Motion-Scenario Decoupling (DMSD) framework. The architecture is designed to prevent the model from being "distracted" by background pixels and instead forces it to understand the relationship between the environment and the trajectory.

Decoupling Strategy

The framework employs a pre-decoupling operator consisting of two modules:

  • Motion Removal Module (MRM): Isolates the static background (the scenario) by removing dynamic elements.
  • Scenario Removal Module (SRM): Isolates the moving subject (the motion) by subtracting the background features.

Dual-stream Architecture

These decoupled streams are processed through parallel Temporal Shift ResNets. This dual-stream approach ensures that the model learns environmental constraints (e.g., walls and obstacles) independently of the rat's specific movement patterns.

Specialized Loss Functions

To refine the learning process, the authors introduced two novel loss functions:

  • Scenario Contrast Loss ($L_{sc}$): Utilizes contrastive learning to increase sensitivity to subtle environmental changes, such as the introduction of new objects or terrain.
  • Motion Clustering Loss ($L_{mc}$): Groups similar movement features to help the model navigate the inherent uncertainty of animal intent.

Experimental Results

The DMSD framework was evaluated against several state-of-the-art (SOTA) video models, including TSM and TRN. The results demonstrate the effectiveness of the decoupling strategy:

  • Accuracy Boost: DMSD achieved a Top-1 Accuracy of 45.63% in single-scenario tasks, representing a 34.3% relative improvement over standard models.
  • Robustness: The model maintained consistent performance across different tasks and showed significantly lower standard deviation than its competitors, even in noisy environments like dense fog.
  • Generalization: DMSD proved more resilient to changes in camera perspective, a common failure point for models that do not explicitly decouple motion from the scene.

Conclusion and Future Work

The DMSD framework and RatPose dataset represent a significant step forward in the control and monitoring of bio-robotic systems. By successfully decoupling motion from scenario, the model provides more reliable predictions of animal trajectories in complex environments.

Future research will focus on scaling the RatPose dataset to include more species and complex "wild" environments. Additionally, the authors aim to integrate these vision-based predictions directly into BCI control loops to enhance the real-time coordination between human operators and bio-robotic agents.

Summary updated

BibTeX
@inproceedings{liu2023motion,
  author = {Xiaofeng Liu and Jiaxin Gao and Yaohua Liu and Nenggan Zheng and Risheng Liu},
  title = {{Motion-Scenario Decoupling for Rat-Aware Video Position Prediction: Strategy and Benchmark}},
  booktitle = {ICIG},
  year = {2023},
  doi = {10.1007/978-3-031-46308-2\_12},
}
Figure