Abstract
In this work, we present a single-stage framework, named S2F2, for forecasting multiple human trajectories from raw video images by predicting future optical flows. S2F2 differs from the previous two-stage approaches in that it performs detection, Re-ID, and forecasting of multiple pedestrians at the same time. Unlike the prior approaches, the computational burden of S2F2 remains consistent even as the number of pedestrians grows. The experimental results demonstrate that S2F2 is able to outperform two conventional forecasting algorithms and a recent learning-based two-stage model, while maintaining its tracking performance on par with the contemporary MOT models.
The Shift to Single-Stage Forecasting
Traditionally, predicting where people will walk in a video has been a "two-stage" problem. First, a model detects and tracks individuals (Stage 1), and then a separate forecasting model takes those tracking histories to predict future paths (Stage 2). This approach suffers from error propagation—where detection failures in the first stage inevitably lead to forecasting failures—and poor computational scaling, as most models must run separate inference passes for every detected person.
S2F2 (Single-Stage Flow Forecasting) addresses these flaws by creating a unified architecture that performs detection, Re-ID, and forecasting simultaneously. This ensures that the computational burden remains consistent regardless of the number of pedestrians in the scene.
The S2F2 Architecture
S2F2 is built atop an anchor-free multiple object tracking (MOT) design and consists of two primary components: the Context Feature Extractor and the Forecasting Module.
Context Feature Extractor
Using a Deep Layer Aggregation (DLA) backbone, the model extracts a shared latent feature embedding. This embedding is multi-purpose, used for detecting bounding boxes, generating re-identification (Re-ID) features, and providing the necessary context for the forecasting task.
Forecasting Module
Rather than predicting individual $(x, y)$ coordinates for each pedestrian, S2F2 utilizes Convolutional Gated Recurrent Units (ConvGRUs) to predict future optical flow maps. These maps represent estimated pixel-wise offsets from the current frame to future frames. By "warping" current detection results forward using these predicted flows, the model can forecast trajectories for every person in the frame at once. This dense flow approach allows the model to implicitly capture social interactions and spatial correlations, such as a crowd moving together to avoid obstacles.
Experimental Evaluation
The authors evaluated S2F2 on StaticMOT, a subset of the MOT17 and MOT20 datasets that excludes sequences with egocentric camera motion. This focus allows for a more controlled evaluation of human trajectory prediction capabilities.
Key Results
- Accuracy: S2F2 achieved an Average Displacement Error (ADE) of 12.275, significantly outperforming the learning-based two-stage model STED (ADE 16.928) and conventional baselines like Kalman Filters.
- Efficiency: The forecasting portion of S2F2 takes approximately 13.8ms regardless of crowd size. In dense scenes, this makes it nearly 45 times faster than the STED model, which required 623.5ms on the same hardware.
- Tracking Quality: Despite the added complexity of forecasting, the model maintained tracking performance (MOTA and IDF1) on par with specialized state-of-the-art models like FairMOT. Furthermore, the forecasting module was used to "rescue" low-confidence detections by checking if they aligned with predicted flow, reducing fragmented trajectories.
Limitations and Future Work
While S2F2 represents a significant step forward, it is currently optimized for static cameras. In dynamic environments where the camera moves (e.g., on a vehicle), the optical flow would be dominated by ego-motion rather than pedestrian intent. Additionally, the model can struggle with sudden, erratic movements or identity switches in extremely dense crowds where object centers overlap in the flow field. Future research will focus on extending this single-stage philosophy to dynamic camera environments and improving robustness for non-linear movements.
Summary updated