End-to-end trainable network for degraded license plate detection via vehicle-plate relation mining

Song-Lu Chen, Shu Tian, Jia-Wei Ma, Qi Liu, Chun Yang, Feng Chen, Xu-Cheng Yin

Abstract

License plate detection (LPD) is a fundamental step in recognition systems but remains challenging in real-world on-road scenarios. Small-sized and oblique license plates, often caused by distant or mobile cameras, are particularly difficult to detect. We propose a novel method for degraded LPD via vehicle-plate relation mining using a coarse-to-fine scheme. First, we estimate the local region around the license plate by leveraging the spatial relationship between the vehicle and the plate, significantly reducing the search area. Second, we predict a quadrilateral bounding box by regressing the four corners of the plate, ensuring robustness against oblique viewpoints. The entire network is end-to-end trainable. Extensive experiments demonstrate the effectiveness of our method for small and tilted license plates.

Introduction

License plate detection is the "front door" of automated traffic systems, yet standard algorithms often fail in the wild. In on-road scenarios, plates are frequently small-sized (occupying less than 0.26% of pixels) or oblique (appearing as tilted quadrilaterals). Standard detectors like SSD or YOLO often miss these tiny features or fail to provide the precise localization required for Optical Character Recognition (OCR). While vehicle detection can narrow the search, existing methods often struggle with large vehicles or require tedious manual labeling. This paper introduces an end-to-end pipeline that mines the spatial relationship between a vehicle and its plate to overcome these hurdles.

Methodology

The proposed architecture moves from global vehicle context to precise plate localization through three distinct stages:

  1. Approximate License Plate Detection (ALPD): Based on an SSD backbone, this stage detects the vehicle and predicts the spatial offset between the vehicle's center and the license plate's center, along with an estimated plate size.
  2. Local Region Estimation and Aggregation (LREA): The network defines a "local region" around the suspected plate. To handle multiple vehicles of different sizes, it uses ROI Warping to crop and scale these regions into uniform "LP patches."
  3. Multi-Oriented License Plate Refinement (MOLPR): Within these patches, the network regresses the four specific corners of the plate. This quadrilateral approach allows the model to tightly bound oblique or distorted plates.
Overview of the proposed end-to-end pipeline, illustrating the transition from vehicle detection to refined plate localization.

Key Insights

  • Contextual Priors: Treating the vehicle-plate distance as a learnable offset provides a powerful spatial prior, making tiny plates easier to find.
  • Scale Invariance: ROI Warping effectively "zooms in" on distant plates, ensuring the final layers have sufficient resolution regardless of vehicle distance.
  • Quadrilateral Regression: Directly regressing four corners is essential for the accuracy required by subsequent OCR steps, as it excludes background noise present in standard rectangular boxes.

Experimental Results

The researchers evaluated the method on two datasets: TILT720 (1,033 images) and TILT1080 (4,112 images), featuring multi-oriented plates in on-road scenes.

The proposed method significantly outperformed standard detectors, especially at higher precision requirements (IOU=0.75). On the TILT720 dataset, the method achieved 54.51% Average Precision (AP), compared to 47.06% for a standard SSD and 13.63% for Faster R-CNN. Qualitative results show the system successfully detecting plates on distant vehicles that were virtually invisible to other models.

Final detection results demonstrating the model's ability to accurately localize license plates using quadrilateral bounding boxes.

Conclusion

By mining vehicle-plate relationships, this end-to-end network provides a robust solution for detecting degraded license plates. The coarse-to-fine approach, combined with quadrilateral regression, addresses the primary challenges of small size and oblique viewpoints. Future work will focus on improving the initial vehicle detection phase to further boost overall system reliability in extreme conditions.

Summary updated

Abstract

License plate detection is the first and essential step of the license plate recognition system and is still challenging in real applications, such as on-road scenarios. In particular, small-sized and oblique license plates, mainly caused by the distant and mobile camera, are difficult to detect. In this work, we propose a novel and applicable method for degraded license plate detection via vehicle-plate relation mining, which localizes the license plate in a coarse-to-fine scheme. First, we propose to estimate the local region around the license plate by using the relationships between the vehicle and the license plate, which can greatly reduce the search area and precisely detect very small-sized license plates. Second, we propose to predict the quadrilateral bounding box in the local region by regressing the four corners of the license plate to robustly detect oblique license plates. Moreover, the whole network can be trained in an end-to-end manner. Extensive experiments verify the effectiveness of our proposed method for small-sized and oblique license plates. Codes are available at https://github.com/chensonglu/LPD-end-to-end.
BibTeX
@inproceedings{chen2021end,
  author = {Song-Lu Chen and Shu Tian and Jia-Wei Ma and Qi Liu and Chun Yang and Feng Chen and Xu-Cheng Yin},
  title = {{End-to-end trainable network for degraded license plate detection via vehicle-plate relation mining}},
  booktitle = {Neurocomputing},
  year = {2021},
  url = {https://openreview.net/forum?id=k3hLhiCTUhJw},
}
Figure