Multi-timescale Trajectory Prediction for Abnormal Human Activity Detection

Royston Rodrigues, Neha Bhargava, Rajbabu Velmurugan, Subhasis ChaudhuriWACV 2020

TL;DR

This paper introduces a novel multi-timescale framework for detecting abnormal human activities by predicting future and past pose trajectories at various temporal granularities. By capturing anomalies that manifest over different durations, their method significantly outperforms existing single-timescale approaches and is validated on a new, challenging dataset.

What Problem Does This Paper Address?

Detecting abnormal human activity in surveillance footage is a critical but challenging task. A major hurdle is that "abnormal" is highly context-dependent and, more importantly, anomalies can occur over vastly different timescales. For instance, loitering is a long-term anomaly, while a sudden jump might be a short-term one. Traditional methods often rely on a single, fixed timescale (e.g., frame-by-frame or fixed-duration clips) to learn normal behavior. This limitation means they can miss anomalies that unfold too slowly or too quickly for their chosen timescale, or misclassify normal but infrequent events as abnormal. The paper aims to bridge this gap by developing a model that can effectively learn and detect anomalies across a spectrum of temporal scales.

Methods

The core of the proposed solution is a multi-timescale prediction framework. This framework consists of two main components: a future prediction model and an identical past prediction model. Both models take a human pose trajectory (represented by 25 keypoints) as input.

The key innovation lies in how predictions are generated at different timescales. The model is structured hierarchically, with intermediate layers designed to produce predictions corresponding to specific temporal scales (e.g., 3, 5, 13, and 25 frames). This is achieved by providing supervision at these intermediate layers during training. For a given input sequence, the model splits it into smaller sub-sequences and makes predictions for these sub-sequences at each timescale. These predictions are then combined to form a prediction for the entire input sequence at that specific timescale.

To detect anomalies, the model calculates prediction errors at each timescale. These errors are then combined using a voting mechanism to produce a final abnormality score for each time instant. The authors also introduce a weighted mean square error loss function, where weights are derived from the confidence scores of the pose detector, to better handle variations in pose estimation accuracy.

Datasets

The authors introduce a new dataset called IITB-Corridor. This dataset is captured from a single camera in a corridor setting and includes a variety of normal activities (walking, standing) and enacted abnormal activities such as protest, unattended baggage, cycling, sudden running, fighting, chasing, loitering, and hiding faces. It contains 1,81,567 test frames with 1,08,278 abnormal frames.

The proposed method is also evaluated on two public datasets:

  • HR-ShanghaiTech Campus dataset: A challenging dataset with multiple cameras and various anomalies like running, fighting, and loitering.
  • HR-Avenue dataset: Another public dataset known for anomalies such as throwing bags, running, and dancing.

Key Insights and Ideas

  • The importance of multi-timescale analysis: The paper strongly demonstrates that a single timescale is insufficient for robust abnormal activity detection. Capturing temporal dynamics across multiple scales is crucial for identifying a wider range of anomalies.
  • Past and future prediction synergy: Combining both past and future predictions, rather than relying solely on future prediction, leads to improved performance. This suggests that understanding the trajectory's history is as important as predicting its future for anomaly detection.
  • Hierarchical prediction for temporal granularity: The novel approach of using intermediate layers in a deep network to generate predictions at different timescales is an effective way to learn multi-scale temporal representations.
  • Dataset diversity matters: The introduction of the IITB-Corridor dataset, with its diverse set of single-person and group anomalies, addresses a need for more challenging benchmarks in this domain.

Key Results

The proposed multi-timescale framework consistently outperforms existing state-of-the-art methods on the evaluated datasets. For instance, on the HR-ShanghaiTech dataset, the proposed method achieves an AUC of 77.04%, surpassing the next best method (Morais et al. [13]) at 75.40%. Similarly, on the HR-Avenue dataset, the proposed method achieves 88.33% AUC, outperforming Morais et al. [13] at 86.30%. The paper also shows that incorporating multiple timescales and past prediction models significantly improves performance, as evidenced by ablation studies.

Limitations and Future Work

The authors acknowledge that their current model primarily focuses on single-person based anomalies. Future work will extend this framework to address anomalies involving multiple persons or group interactions. While the IITB-Corridor dataset is designed to be challenging, it is still a single-camera dataset, and future work might explore multi-camera scenarios. The paper also notes that the model doesn't capture non-human anomalies (e.g., a car).

Summary updated

BibTeX
@inproceedings{rodrigues2020multi,
  author = {Royston Rodrigues and Neha Bhargava and Rajbabu Velmurugan and Subhasis Chaudhuri},
  title = {{Multi-timescale Trajectory Prediction for Abnormal Human Activity Detection}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure