Scale-Invariant Multidirectional License Plate Detection with the Network Combining Indirect and Direct Branches

Song-Lu Chen, Qi Liu, Jia-Wei Ma, Chun YangSensors 2021 · DOI

TL;DR

This paper introduces a novel end-to-end deep learning network for license plate detection (LPD) that combines two complementary branches: an "indirect" branch leveraging vehicle-plate relationships for precise detection of small plates, and a "direct" branch to catch plates missed by the indirect method. This dual-branch approach significantly improves detection accuracy, especially for challenging multi-scale and multi-directional license plates in real-world scenarios.

What Problem Does This Paper Address?

Detecting license plates in real-world images is a notoriously difficult task due to several factors:

  • Scale Variation: License plates can appear very small in an image, making them hard to distinguish from the background.
  • Orientation: Plates are often not perfectly horizontal due to camera angles and vehicle positioning, appearing tilted or even at arbitrary angles.
  • Environmental Conditions: Variations in lighting, shadows, and occlusions further complicate detection.

Existing methods often fall into two categories: "direct" methods that try to find the plate directly in the image, and "indirect" methods that first detect the vehicle and then search for the plate within the vehicle's region. Direct methods struggle with small plates, while indirect methods can fail if the vehicle itself isn't detected. This paper aims to overcome these limitations by creating a robust system that leverages the strengths of both approaches.

Methods

The core innovation is a novel network architecture that integrates two distinct detection branches into a single, end-to-end trainable system:

  1. Indirect Detection Branch: This branch uses a coarse-to-fine strategy.
  2. Approximate License Plate Detection (ALPD): It first detects vehicles and then estimates the approximate location and size of the license plate based on the spatial relationship (offset) between the vehicle's center and the plate's center. This stage also predicts if a vehicle contains a license plate.
  3. Local Region Estimation and Aggregation (LREA): The approximate plate region is then expanded to create a local region of interest, which is aggregated into feature patches.
  4. Multidirectional License Plate Refinement (MLPR): Within this local region, the network refines the detection by predicting both a horizontal bounding box and, crucially, the four corners of the license plate. This corner-based prediction allows for accurate localization of arbitrarily oriented plates.
  5. Direct Detection Branch (DLPD): This branch operates directly on the input image, similar to standard object detectors. Its primary role is to act as a fallback, detecting license plates that might have been missed by the indirect branch (e.g., due to vehicle detection failures).

Integration: The outputs from both branches are merged using post-processing techniques like Non-Maximum Suppression (NMS). The entire network is trained end-to-end, allowing all components to learn and optimize collaboratively.

Datasets

The authors utilized two custom datasets for training and evaluation:

  • TILT720: Collected from a driving recorder, featuring road scenes with a resolution of 720x1280. It includes annotated vehicles (horizontal bounding boxes) and license plates (quadrilateral bounding boxes).
  • TILT1080: Similar to TILT720 but with higher resolution (1080x1920) and a larger number of images.

These datasets were randomly split into training-validation (9:1) and test sets.

Key Insights and Ideas

  • Complementary Strengths: The paper highlights that indirect and direct detection methods are not mutually exclusive but rather complementary. Combining them effectively addresses the weaknesses of each individual approach.
  • Vehicle-Plate Relationship is Key for Small Plates: The indirect branch's reliance on vehicle context proves highly effective for detecting small license plates, which are often lost in direct detection methods.
  • Corner Prediction for Multidirectionality: Instead of complex rotation estimation, predicting the four corners of the license plate offers a simpler yet effective way to handle arbitrary orientations.
  • End-to-End Learning Advantage: Training the entire network end-to-end allows for joint optimization, ensuring that the indirect and direct branches learn to work together harmoniously.

Key Results

The proposed "Two-branch" method demonstrated superior performance across various metrics and datasets:

  • Horizontal Bounding Box AP: Achieved the best Average Precision (AP) on both TILT720 and TILT1080 datasets for both IoU=0.5 and IoU=0.75 thresholds, outperforming methods like Faster R-CNN, YOLOv2, and SSD. For instance, on TILT720 with IoU=0.75, the two-branch method reached 56.54% AP, significantly better than SSD (47.06%) and Method [17] (54.51%).
  • Multidirectional License Plate F₁-Score: For quadrilateral bounding box detection, the two-branch method achieved the highest F₁-scores, particularly at higher IoU thresholds. On TILT1080 (IoU=0.75), it reached 62.55%, compared to SSD (43.08%) and Method [17] (61.42%).
  • Small-Sized License Plate Recall: The method showed a significant improvement in Recall for small license plates, with near-perfect recall (e.g., 77.89% on TILT720, IoU=0.5/0.75) compared to other methods.

Qualitative results (Figure 9) show successful detection of challenging plates, with the two-branch method producing more complete and accurate results than either branch alone.

Limitations and Future Work

The authors acknowledge several limitations:

  • Failure Cases: Despite improvements, the method can still fail in extreme conditions, such as severe illumination variations (mirror reflections), vehicle occlusion, or significant deviations in vehicle-plate offset.
  • Post-processing Impact: In some cases, the post-processing module (NMS) might remove true-positive predictions, leading to a slight drop in recall.

Future work could focus on:

  • Improving Vehicle Detection: Enhancing the vehicle detection component to better handle occlusions and boundary truncations would directly benefit the indirect branch.
  • Robustness to Extreme Conditions: Developing more robust mechanisms to handle severe illumination variations and large orientation deviations.
  • Optimizing Post-processing: Investigating alternative post-processing strategies to minimize the loss of true positives.

Summary updated

Abstract

As the license plate is multiscale and multidirectional in the natural scene image, its detection is challenging in many applications. In this work, a novel network that combines indirect and direct branches is proposed for license plate detection in the wild. The indirect detection branch performs small-sized vehicle plate detection with high precision in a coarse-to-fine scheme using vehicle-plate relationships. The direct detection branch detects the license plate directly in the input image, reducing false negatives in the indirect detection branch due to the miss of vehicles' detection. We propose a universal multidirectional license plate refinement method by localizing the four corners of the license plate. Finally, we construct an end-to-end trainable network for license plate detection by combining these two branches via post-processing operations. The network can effectively detect the small-sized license plate and localize the multidirectional license plate in real applications. To our knowledge, the proposed method is the first one that combines indirect and direct methods into an end-to-end network for license plate detection. Extensive experiments verify that our method outperforms the indirect methods and direct methods significantly.
BibTeX
@inproceedings{chen2021scale,
  author = {Song-Lu Chen and Qi Liu and Jia-Wei Ma and Chun Yang},
  title = {{Scale-Invariant Multidirectional License Plate Detection with the Network Combining Indirect and Direct Branches}},
  booktitle = {Sensors},
  year = {2021},
  doi = {10.3390/s21041074},
}
Figure