Abstract
Using offline training schemes, researchers have tackled the event segmentation problem by providing full or weak-supervision through manually annotated labels or self-supervised epoch-based training. Most works consider videos that are at most 10's of minutes long. We present a self-supervised perceptual prediction framework capable of temporal event segmentation by building stable representations of objects over time and demonstrate it on long videos, spanning several days. The approach is deceptively simple but quite effective. We rely on predictions of high-level features computed by a standard deep learning backbone. For prediction, we use an LSTM, augmented with an attention mechanism, trained in a self-supervised manner using the prediction error. The self-learned attention maps effectively localize and track the event-related objects in each frame. The proposed approach does not require labels. It requires only a single pass through the video, with no separate training set. Given the lack of datasets of very long videos, we demonstrate our method on video from 10 days (254 hours) of continuous wildlife monitoring data that we had collected with required permissions. We find that the approach is robust to various environmental conditions such as day/night conditions, rain, sharp shadows, and windy conditions. For the task of temporally locating events, we had an 80% recall rate at 20% false-positive rate for frame-level segmentation. At the activity level, we had an 80% activity recall rate for one false activity detection every 50 minutes. We will make the dataset, which is the first of its kind, and the code available to the research community.
Introduction
Wildlife researchers often deploy cameras that record continuously for days or weeks, resulting in "extended videos" that are impossible to annotate manually. Traditional computer vision approaches to event segmentation usually rely on offline training, where a model must see a labeled dataset multiple times (epochs) before it can be used. This creates two major hurdles: the prohibitive cost of manual labeling for hundreds of hours of footage, and the "stream challenge," where models must learn and infer simultaneously to adapt to specific camera environments.
This paper presents a framework for spatio-temporal localization—identifying both when an event happens and where in the frame it is occurring—using a purely self-supervised, single-pass approach. The system is validated on a novel 254-hour dataset of Kagu bird monitoring in New Caledonia, featuring complex environmental noise such as IR night transitions, heavy rain, and moving shadows.
Methodology
The proposed framework is inspired by cognitive psychology: the idea that humans perceive "event boundaries" when internal predictions about the immediate future fail.
The architecture consists of three main stages:
- Feature Encoding: Raw video frames are passed through a frozen Inception V3 backbone (pre-trained on ImageNet) to extract high-level spatial features.
- Attention-Augmented Prediction: The core of the model is a Long Short-Term Memory (LSTM) network equipped with a Bahdanau attention mechanism. Instead of classifying the video, the LSTM predicts the next frame’s features based on the current state.
- Self-Supervised Learning: The model is trained "on the fly" using the prediction error. If the difference between the predicted features and the actual next frame is high, the system signals a potential event.
To improve performance, the authors introduce a Motion-Weighted Loss that forces the model to focus on moving objects rather than static backgrounds, and an Adaptive Thresholding mechanism to maintain robustness across changing lighting and weather conditions.
Results and Discussion
The model was evaluated on its ability to flag frames containing bird activity (e.g., feeding, incubation, nest building). It achieved an 80% recall rate at a 20% false-positive rate for frame-level segmentation. At the activity level, it reached 80% recall with only one false detection every 50 minutes.
Key insights from the results include:
- Attention as a Localization Tool: The attention maps effectively "lock on" to the bird to help the LSTM predict the next frame. This allows the model to spatially localize events without ever being explicitly trained on bird images.
- Learning by Failing: The model identifies events by detecting deviations from the learned "normal" state. When a bird enters a static nest, the model’s prediction fails, triggering a detection.
- Single-Pass Efficiency: Because training and inference happen simultaneously in a single pass, the model is highly suitable for real-time monitoring on edge devices with limited storage.
Limitations and Future Work
While effective, the prediction-error approach flags any significant change, meaning it cannot currently distinguish between different types of novel events (e.g., a predator vs. a chick) without additional logic. Furthermore, very slow-moving objects may eventually be incorporated into the model's "predicted background." Future research could explore integrating weak supervision or multi-camera setups to refine event categorization.
Summary updated