Turning Traffic Monitoring Cameras into Intelligent Sensors for Traffic Density Estimation

Zijian Hu, William H. K. Lam, Sze Chun Wong, Andy H. F. Chow, Wei MaarXiv 2021 · OpenReview

TL;DR

This paper introduces a novel framework to transform existing traffic monitoring cameras into intelligent sensors for estimating traffic density. By addressing the inherent challenges of low-resolution, low frame rate, and lack of annotated data, the proposed method achieves accurate traffic density estimation, paving the way for more efficient traffic management systems without requiring new hardware.

What Problem Does This Paper Address?

Intelligent Transportation Systems (ITS) rely heavily on accurate traffic state information (speed, flow, density) for applications like traffic signal coordination and flow prediction. Traditionally, this data comes from specialized sensors like loop detectors. However, many cities already have extensive networks of traffic monitoring cameras used for visual inspection. The challenge is that data from these cameras suffers from "4L characteristics": Low frame rate, Low resolution, Lack of annotated data, and Located in complex road environments. These limitations make it difficult to extract reliable traffic state information. This paper specifically tackles the problem of traffic density estimation from these challenging camera feeds, aiming to leverage existing infrastructure rather than requiring costly new sensor deployments.

Methods

The proposed framework has two core components:

  1. Camera Calibration (MVCalib): This novel method addresses the "4L" challenges to accurately determine the real-world length represented by pixels in camera images. It uses a multi-vehicle approach, leveraging key points from multiple vehicles simultaneously to improve robustness. The method proceeds in three stages:
  2. Candidate Generation: Initial camera parameter estimates are generated using conventional methods for each vehicle.
  3. Vehicle Model Matching: The best-fit vehicle model is identified for each vehicle to refine parameter estimates.
  4. Parameter Fine-tuning: Joint information from multiple vehicles is used to fine-tune the camera parameters (focal length, rotation, translation) for optimal accuracy. This stage is crucial for handling the unknown focal length and low-resolution images.
  5. Vehicle Detection: A deep learning-based approach is employed to count vehicles. To overcome the "lack of annotated data" challenge, a linear-program-based data mixing strategy is introduced. This strategy hybridizes multiple public traffic datasets (e.g., BDD100K, UA-DETRAC) to create a balanced dataset that covers various traffic scenarios (daytime/nighttime, different conditions). This "LP hybrid dataset" is then used to train a YOLO-v5 model, enabling robust vehicle detection even with limited annotations for specific camera systems.

Datasets

The paper utilizes several public datasets for training the vehicle detection model:

  • BDD100K: Large-scale dataset for autonomous driving.
  • BITVehicle: Used for vehicle re-identification.
  • CityCam: Focused on vehicle detection.
  • COCO: General object detection dataset.
  • MIO-TCD-L: For vehicle detection and classification.
  • UA-DETRAC: Primarily for vehicle detection.

Additionally, the paper uses data from two real-world traffic monitoring systems for case studies:

  • Hong Kong (HK) eMobility System: Data from Chatham Road South, Kowloon.
  • Caltrans System (Sacramento, CA): Data from the I-50 Highway.

Ground truth data for camera calibration was derived from field measurements of road markings, and for traffic density, it was obtained from double-loop detectors in Sacramento and a high-resolution ground truth camera in Hong Kong.

Key Insights and Ideas

  • Leveraging Existing Infrastructure: The core idea is to repurpose ubiquitous traffic cameras, which are often underutilized for quantitative analysis, into valuable traffic sensors. This avoids the cost and complexity of installing new specialized hardware.
  • Multi-Vehicle Calibration for Robustness: By considering multiple vehicles simultaneously in the calibration process, the method becomes more robust to the low-resolution and potential inaccuracies of individual vehicle key points. This is a significant departure from methods relying on single-vehicle models.
  • Data Hybridization for Generalization: The linear programming approach to create a balanced "LP hybrid dataset" is key to training a vehicle detection model that generalizes well across different traffic conditions (day/night) and camera systems, mitigating the issue of limited annotated data for specific surveillance setups.
  • Addressing the "4L" Challenges Holistically: The paper presents a unified framework that tackles all four "L" challenges (Low frame rate, Low resolution, Lack of annotated data, Complex road environments) in a coordinated manner, rather than addressing them in isolation.

Key Results

  • Camera Calibration: The proposed MVCalib method achieved a Mean Absolute Error (MAE) of less than 0.2 meters out of 6 meters for road length estimation.
  • Vehicle Detection: The detection model achieved an accuracy of approximately 90% under various conditions.
  • Traffic Density Estimation:
    • In Hong Kong, the MAE for estimated traffic density was 9.04 veh/km/lane.
  • In Sacramento, the MAE was 1.30 veh/km/lane.
  • The framework demonstrated good performance in both case studies, indicating its generalizability across different locations and camera systems.
  • The model trained on the LP hybrid dataset showed improved performance at nighttime compared to a model trained on a combined, unbalanced dataset.

Limitations and Future Work

  • Performance Degradation in Congestion: The authors note that the performance of the density estimation framework degrades under highly congested traffic conditions.
  • Frame Rate Limitations: The low frame rate of some cameras (e.g., 2 minutes per image in Hong Kong) can lead to biased estimations, especially when dealing with small sample sizes and non-recurrent traffic patterns.
  • Lane Determination Bias: The method for determining vehicle lanes based on bounding box centers can be biased on curved roads or with large vehicles.
  • Future Work:
    • Extend the framework to estimate other traffic state variables like speed, flow, and occupancy.
  • Automate the key point annotation process for camera calibration.
  • Develop a vehicle classification model to estimate traffic density by vehicle type.
  • Explore domain adaptation techniques for improved detection across diverse traffic scenarios.
  • Develop a fully automated, end-to-end pipeline for deployment.

Summary updated

Abstract

Accurate traffic state information plays a pivotal role in the Intelligent Transportation Systems (ITS), and it is an essential input to various smart mobility applications such as signal coordination and traffic flow prediction. The current practice to obtain the traffic state information is through specialized sensors such as loop detectors and speed cameras. In most metropolitan areas, traffic monitoring cameras have been installed to monitor the traffic conditions on arterial roads and expressways, and the collected videos or images are mainly used for visual inspection by traffic engineers. Unfortunately, the data collected from traffic monitoring cameras are affected by the 4L characteristics: Low frame rate, Low resolution, Lack of annotated data, and Located in complex road environments. Therefore, despite the great potentials of the traffic monitoring cameras, the 4L characteristics hinder them from providing useful traffic state information (e.g., speed, flow, density). This paper focuses on the traffic density estimation problem as it is widely applicable to various traffic surveillance systems. To the best of our knowledge, there is a lack of the holistic framework for addressing the 4L characteristics and extracting the traffic density information from traffic monitoring camera data. In view of this, this paper proposes a framework for estimating traffic density using uncalibrated traffic monitoring cameras with 4L characteristics. The proposed framework consists of two major components: camera calibration and vehicle detection. The camera calibration method estimates the actual length between pixels in the images and videos, and the vehicle counts are extracted from the deep-learning-based vehicle detection method. Combining the two components, high-granular traffic density can be estimated. To validate the proposed framework, two case studies were conducted in Hong Kong and Sacramento. The results show that the Mean Absolute Error (MAE) in camera calibration is less than 0.2 meters out of 6 meters, and the accuracy of vehicle detection under various conditions is approximately 90%. Overall, the MAE for the estimated density is 9.04 veh/km/lane in Hong Kong and 1.30 veh/km/lane in Sacramento. The research outcomes can be used to calibrate the speed-density fundamental diagrams, and the proposed framework can provide accurate and real-time traffic information without installing additional sensors.
BibTeX
@misc{hu2021turning,
  author = {Zijian Hu and William H. K. Lam and Sze Chun Wong and Andy H. F. Chow and Wei Ma},
  title = {{Turning Traffic Monitoring Cameras into Intelligent Sensors for Traffic Density Estimation}},
  year = {2021},
  eprint = {2111.00941},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.00941},
}
Figure