Abstract
Traffic Anomaly detection is an essential computer vision task and plays a critical role in video structure analysis and urban traffic analysis. In this paper, we propose a box-level tracking and refinement algorithm to identify anomaly detection in road scenes. We first link the detection results to construct candidate spatio-temporal tubes via greedy search. Then the box-level refinement scheme is introduced to employ auxiliary detection cues to promote the abnormal predictions, which consists of spatial fusion, still-thing filter, temporal fusion, and feedforward optimization. Still-thing filter and feedforward optimization employ complementary detection concepts to promote the abnormal predictions, which helps determine an accurate abnormal period. The experimental results show that our approach is superior in the Traffic Anomaly Detection Track test set of the NVIDIA AI CITY 2021 CHALLENGE, which ranked second in this competition, with a 93.18% F1-score and 3.1623 root mean square error. It reveals that the proposed approach contributes to fine-grained anomaly detection in actual traffic accident scenarios and promoting the development of intelligent transportation.
Introduction
Traffic anomaly detection—identifying accidents, stalled vehicles, or illegal maneuvers—is a cornerstone of "Smart City" infrastructure. However, automating this process remains difficult due to two major hurdles:
- The "Still-Thing" Problem: Automated systems often struggle to distinguish between a vehicle that just crashed (an anomaly) and static background objects like road signs, trees, or legally parked cars.
- Temporal Precision: Background modeling techniques often have a "lag." It takes time for a system to realize a vehicle has stopped rather than being stuck in heavy traffic, making it hard to identify the exact second an accident occurred.
This research addresses these challenges by focusing on the assumption that any vehicle stopped longer than a typical traffic light cycle constitutes an abnormal event.
Methodology
The proposed framework consists of four major stages: pre-processing, vehicle detection, box-level tube tracking, and a multi-stage refinement scheme.
Pre-processing: Differential Masks and Background Modeling
To ignore irrelevant areas such as sidewalks or parking lots, the system uses Differential Masks based on video motion to segment only the main roadway. This ensures the detector focuses on areas where traffic anomalies are most critical. Following mask extraction, the system employs Backward Background Modeling using a Gaussian Mixture Model (GMM). By processing the video in reverse, moving traffic is effectively "cleared out," making motionless vehicles much easier for the detector to isolate.
Vehicle Detection and Tube Tracking
The system utilizes a Cascade R-CNN (with a ResNeXt-101 backbone) to detect vehicles. This model is run on both the original video frames and the frames generated by the background model. Detections are then linked across frames using a greedy search algorithm based on Intersection-over-Union (IoU) scores. This creates "spatio-temporal tubes"—3D paths representing a vehicle's presence and position over time.
The Box-Level Refinement Scheme
The core innovation of this work is the refinement pipeline, which consists of four filters designed to polish the candidate tubes:
- Spatial Fusion: Merges overlapping tubes that likely represent the same vehicle.
- Still-thing Filter: Compares detections from the background model against the original video. If an object (like a sign) was present throughout the entire video, it is discarded as a "still-thing" rather than an anomaly.
- Temporal Fusion: Merges tubes that are close together in time to ensure a single event isn't split into multiple alerts.
- Feedforward Optimization: To fix the "lag" in background modeling, the system looks back at original frames prior to the detected stop to find the true moment the vehicle became stationary.
Experimental Results
The model was evaluated on the NVIDIA AI CITY 2021 CHALLENGE Track 4 dataset, which includes 100 training and 150 test videos of diverse highway and intersection scenarios. The approach achieved highly competitive results:
- F1-Score: 93.18% (Ranked 2nd in the competition).
- RMSE (Timing Error): 3.1623 seconds.
- Total Score (S4): 0.9220.
The results demonstrate that the refinement scheme—specifically the still-thing filter—significantly reduces false positives that plague simpler detection models.
Discussion and Key Insights
- Reverse Modeling: Running background modeling backward in time is a counter-intuitive but effective way to isolate vehicles that eventually stop, as it effectively "erases" the lead-up traffic.
- Original Video as Ground Truth: While background models are useful, the original video remains the best tool for filtering out "false" anomalies like road signs that the background model might accidentally absorb.
- Correcting Temporal Lag: The "Feedforward" step is essential for achieving high scores in Root Mean Square Error (RMSE), which measures the timing accuracy of the detection.
Limitations
While optimized for stalled vehicles, the framework may be less effective for "dynamic" anomalies, such as high-speed red-light violations where the vehicle does not stop. Additionally, the multi-stage pipeline (Cascade R-CNN, GMM, and refinement) is computationally expensive, which may limit real-time deployment on low-power edge devices. Future work may involve integrating trajectory or velocity cues to handle a broader range of traffic anomalies.
Summary updated