Spatio-Temporal Event Segmentation and Localization for Wildlife Extended Videos

Ramy Mounir, Roman Gula, Jörn Theuerkauf, Sudeep Sarkar

Abstract

Using offline training schemes, researchers have tackled the event segmentation problem by providing full or weak-supervision through manually annotated labels or self-supervised epoch-based training. Most works consider videos that are at most 10's of minutes long. We present a self-supervised perceptual prediction framework capable of temporal event segmentation by building stable representations of objects over time and demonstrate it on long videos, spanning several days. The approach is deceptively simple but quite effective. We rely on predictions of high-level features computed by a standard deep learning backbone. For prediction, we use an LSTM, augmented with an attention mechanism, trained in a self-supervised manner using the prediction error. The self-learned attention maps effectively localize and track the event-related objects in each frame. The proposed approach does not require labels. It requires only a single pass through the video, with no separate training set. Given the lack of datasets of very long videos, we demonstrate our method on video from 10 days (254 hours) of continuous wildlife monitoring data that we had collected with required permissions. We find that the approach is robust to various environmental conditions such as day/night conditions, rain, sharp shadows, and windy conditions. For the task of temporally locating events, we had an 80% recall rate at 20% false-positive rate for frame-level segmentation. At the activity level, we had an 80% activity recall rate for one false activity detection every 50 minutes. We will make the dataset, which is the first of its kind, and the code available to the research community.

Introduction

Wildlife monitoring generates massive amounts of video data, often running continuously for days or weeks. Traditional computer vision approaches to event segmentation typically rely on manual labels or short clips, both of which are impractical for this scale. Annotating hundreds of hours of video is prohibitively expensive, and models trained on short clips often fail to account for long-term environmental shifts like lighting or weather changes. This paper introduces a self-supervised framework for segmenting and localizing events in ultra-long wildlife videos without requiring manual labels or offline training. By using a "perceptual prediction" approach—where the model learns to predict future frames and flags "events" when prediction error spikes—the system achieves high recall in challenging outdoor environments using only a single pass through the data.

Methodology

The proposed framework is inspired by cognitive psychology, which suggests that humans perceive event boundaries when their internal predictions about the immediate future fail. The system processes video in a single pass, learning and inferring simultaneously.

The system architecture includes an Inception V3 encoder, an attention-augmented LSTM, and a self-supervised loss based on prediction error.

Raw video frames are passed through a pre-trained, frozen Inception V3 backbone to extract high-level semantic features, avoiding the noise of low-level motion detection. A Bahdanau-style attention unit spatially localizes the most relevant parts of a frame, allowing the model to track objects even when they are stationary. A Predictive LSTM takes these features and attention maps to predict the features of the next frame. The model is trained on the fly using the prediction error. A key novelty is the Motion Weighted Loss, which incorporates a motion-dependent mask to stabilize attention maps, ensuring the model stays focused on objects of interest even when they stop moving.

Experimental Results

The model was evaluated on the Kagu Bird Dataset, which consists of 10 days (254 hours) of continuous monitoring of a Kagu nest in New Caledonia. This dataset presents significant challenges, including day/night transitions, heavy rain, and sharp moving shadows.

Comparison of prediction loss and motion-weighted loss over time, showing how spikes in error correspond to correct activity detections.

The model achieved an 80% recall rate at a 20% false-positive rate for frame-level segmentation. At the activity level, it reached an 80% activity recall rate with only one false detection every 50 minutes. Qualitative analysis of the attention maps showed successful localization of birds in both high-contrast daytime shadows and low-light infrared night footage, proving the model's robustness across the full diurnal cycle.

Conclusion

This work demonstrates that prediction error can serve as a robust proxy for event detection in complex natural environments without the need for manual labels. By operating in a single pass, the model is viable for deployment on edge devices in the field. Future work will focus on improving event specificity—distinguishing between different types of events— and optimizing the computational load for lower-power field hardware.

Summary updated

Abstract

Using offline training schemes, researchers have tackled the event segmentation problem by providing full or weak-supervision through manually annotated labels or self-supervised epoch-based training. Most works consider videos that are at most 10's of minutes long. We present a self-supervised perceptual prediction framework capable of temporal event segmentation by building stable representations of objects over time and demonstrate it on long videos, spanning several days. The approach is deceptively simple but quite effective. We rely on predictions of high-level features computed by a standard deep learning backbone. For prediction, we use an LSTM, augmented with an attention mechanism, trained in a self-supervised manner using the prediction error. The self-learned attention maps effectively localize and track the event-related objects in each frame. The proposed approach does not require labels. It requires only a single pass through the video, with no separate training set. Given the lack of datasets of very long videos, we demonstrate our method on video from 10 days (254 hours) of continuous wildlife monitoring data that we had collected with required permissions. We find that the approach is robust to various environmental conditions such as day/night conditions, rain, sharp shadows, and windy conditions. For the task of temporally locating events, we had an 80% recall rate at 20% false-positive rate for frame-level segmentation. At the activity level, we had an 80% activity recall rate for one false activity detection every 50 minutes. We will make the dataset, which is the first of its kind, and the code available to the research community.
BibTeX
@misc{mounir2020spatio,
  author = {Ramy Mounir and Roman Gula and Jörn Theuerkauf and Sudeep Sarkar},
  title = {{Spatio-Temporal Event Segmentation and Localization for Wildlife Extended Videos}},
  year = {2020},
  eprint = {2005.02463},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2005.02463},
}
Figure