Zero-VIRUS*: Zero-shot Vehicle Route Understanding System for Intelligent Transportation

Lijun Yu, Qianyu Feng, Yijun Qian, Wenhe Liu, Alexander G. Hauptmann

Introduction

Understanding city-scale traffic flow is a cornerstone of modern intelligent transportation systems. However, automating vehicle counting by class and movement (e.g., distinguishing a left turn from a straight path) at complex intersections remains a significant challenge. Traditional supervised learning approaches are often hindered by the high cost of annotating vehicle trajectories for every unique camera angle and the difficulty of generalizing across diverse environmental conditions and camera distortions.

To address these issues, we present Zero-VIRUS (Zero-shot VehIcle Route Understanding System). This system eliminates the need for human-annotated trajectory data by modeling predefined traffic routes as continuous 2D fields, allowing for robust, zero-shot route identification.

System Architecture

The Zero-VIRUS pipeline operates in two primary stages: high-precision vehicle detection and tracking, followed by mathematical route modeling using spatial fields.

Overview of the Zero-VIRUS pipeline, illustrating the transition from predefined routes to 2D field modeling and final vehicle counting.

1. Detection and Trajectory Enhancement

The system utilizes a Mask R-CNN with a ResNeXt-101 backbone for robust object detection. To mitigate issues with overlapping detections—such as a car and truck being identified in the same spatial region—a Weighted Inter-class Non-Maximum Suppression (NMS) is employed, prioritizing "Car" detections to prevent double-counting.

A key innovation in the tracking phase is the shift from bounding box centers to segmentation-based localization. Standard bounding box centers often "float" in 2D space due to perspective, leading to inaccurate ground-plane trajectories. Zero-VIRUS instead extracts the vehicle's segmentation mask to identify the precise point where the tires contact the road. These paths are further refined using 1D-Gaussian smoothing and linear interpolation to maintain continuity during brief occlusions.

Refinement of vehicle localization by tracking the tire-contact point (G) rather than the bounding box center (A), ensuring more accurate ground-plane trajectories.

2. Route Modeling and Identification

Rather than training a classifier on specific movement patterns, Zero-VIRUS treats each possible route as a mathematical polyline converted into a 2D Field. This "zero-shot" approach evaluates vehicle tracklets against three core metrics:

  • Proximity Field: Measures the spatial distance between the vehicle and the route, normalized by the vehicle's visual scale to account for perspective distortion.
  • Completeness Field: Quantifies how much of a predefined route was traversed, filtering out "ghost" detections or partial movements.
  • Stability Metric: Assesses whether the vehicle moves parallel to the route at a consistent distance, effectively filtering out erratic lane changes.

The final route classification is determined by a weighted aggregation of these three scores, providing a robust mechanism for movement categorization without scene-specific training.

Experimental Results

The system was evaluated on the AIC20_track dataset from the 2020 AI City Challenge. Zero-VIRUS achieved 3rd place overall with a score of 0.9292. Ablation studies confirmed the importance of the multi-metric approach; using all three fields (Proximity, Completeness, and Stability) yielded a significantly higher effectiveness score (0.9554) compared to using proximity alone (0.8903).

In terms of efficiency, the system processes approximately 9 frames per second on a single Nvidia RTX 2080Ti, demonstrating its potential for scalable, real-world deployment.

Conclusion

Zero-VIRUS demonstrates that high-accuracy vehicle route understanding can be achieved without expensive trajectory annotation. By leveraging segmentation-based tracking and 2D field modeling, the system provides a generalizable solution for complex urban intersections. Future work will focus on improving resilience to extreme weather conditions and further automating the definition of route polylines.

Summary updated

Abstract

Nowadays, understanding the traffic statistics in real city-scale camera networks takes an important place in the intelligent transportation field. Recently, vehicle route understanding brings a new challenge to the area. It aims to measure the traffic density by identifying the route of each vehicle in traffic cameras. This year, the AI City Challenge holds a competition with real-world traffic data on vehicle route understanding, which requires both efficiency and effectiveness. In this work, we propose Zero-VIRUS, a Zero-shot VehIcle Route Understanding System, which requires no annotation for vehicle tracklets and is applicable for the changeable real-world traffic scenarios. It adopts a novel 2D field modeling of pre-defined routes to estimate the proximity and completeness of each track. The proposed system has achieved third place on Dataset A in stage 1 of the competition (Track 1: Vehicle Counts by Class at Multiple Intersections) against world-wide participants on both effectiveness and efficiency, with a record of the top place on 50% of the test set.
BibTeX
@inproceedings{yu2020zero,
  author = {Lijun Yu and Qianyu Feng and Yijun Qian and Wenhe Liu and Alexander G. Hauptmann},
  title = {{Zero-VIRUS*: Zero-shot Vehicle Route Understanding System for Intelligent Transportation}},
  booktitle = {Workshops @ CVPR},
  year = {2020},
  doi = {10.1109/cvprw50498.2020.00305},
}
Figure