A Robust Real-Time Automatic License Plate Recognition Based on the YOLO Detector

Rayson Laroca, Evair Severo, Luiz A. Zanlorensi, Luiz S. Oliveira, Gabriel Resende Gonçalves, William Robson Schwartz, David MenottiIJCNN 2018 · DOI · OpenReview

Abstract

Automatic License Plate Recognition (ALPR) has been a frequent topic of research due to many practical applications. However, many of the current solutions are still not robust in real-world situations, commonly depending on many constraints. This paper presents a robust and efficient ALPR system based on the state-of-the-art YOLO object detector. The Convolutional Neural Networks (CNNs) are trained and fine-tuned for each ALPR stage so that they are robust under different conditions (e.g., variations in camera, lighting, and background). Specially for character segmentation and recognition, we design a two-stage approach employing simple data augmentation tricks such as inverted License Plates (LPs) and flipped characters. The resulting ALPR approach achieved impressive results in two datasets. First, in the SSIG dataset, our system achieved a recognition rate of 93.53% and 47 Frames Per Second (FPS), performing better than both Sighthound and OpenALPR commercial systems. Second, targeting a more realistic scenario, we introduce a larger public dataset, called UFPR-ALPR dataset, where our system achieved a recognition rate of 78.33% and 35 FPS, significantly outperforming commercial alternatives.

Introduction

While Automatic License Plate Recognition (ALPR) is a mature field, most existing systems are "brittle," relying on strict constraints such as static cameras, specific viewing angles, or ideal lighting. When these systems encounter "in-the-wild" conditions—such as moving cameras, varied vehicle types (like motorcycles and buses), or tilted plates—their accuracy often plummets. Furthermore, many Deep Learning approaches suffer from a lack of diverse training data. The authors set out to build a system robust enough for real-time use on mobile cameras while handling the complexities of urban traffic.

Methodology

The proposed system utilizes a multi-stage pipeline where each step is powered by a version of the YOLO (You Only Look Once) detector, specifically tuned for the task.

Vehicle and License Plate Detection

The system first identifies the vehicle and then locates the license plate (LP) within that vehicle's bounding box. The authors tested both Fast-YOLO for speed and YOLOv2 for higher accuracy. This hierarchical approach ensures that the LP search space is constrained to relevant regions of the image.

Vehicle and license plate detection using the YOLO-based pipeline.

Character Segmentation and Recognition

Instead of traditional image processing, a modified YOLO-based architecture called CR-NET is used to find and isolate individual characters within the LP patch. To improve recognition accuracy, the process is split into two specialized CNNs: one for letters and one for digits. This leverages the known format of Brazilian LPs (3 letters followed by 4 digits) to reduce confusion between similar-looking characters like '0' and 'O'.

Segmented license plates demonstrating the clarity of character segmentation achieved by the system.

Data Augmentation and Temporal Redundancy

To handle different plate colors (e.g., gray for private cars, red for buses), the authors used "inverted" LPs and flipped characters during training. Furthermore, since the system processes video, it tracks a vehicle across multiple frames and uses a majority voting scheme to decide on the final license plate string, filtering out transient errors.

Datasets and Experimental Results

The study utilizes two primary benchmarks:

  1. SSIG SegPlate Database: A standard dataset of 2,000 frames featuring a static camera and consistent backgrounds.
  2. UFPR-ALPR Dataset: A new, more challenging dataset introduced by the authors. It contains 4,500 frames from 150 videos captured by moving cameras, including diverse vehicle types (cars, motorcycles, buses, trucks) and complex backgrounds.

On the SSIG dataset, the system achieved a 93.53% recognition rate at 47 FPS, outperforming commercial systems like Sighthound (89.80%) and OpenALPR (93.03%). In the more difficult UFPR-ALPR "moving camera" scenario, commercial systems struggled (below 70%), while the proposed system maintained a 78.33% recognition rate at 35 FPS.

Conclusion

The use of YOLO across all stages of the ALPR pipeline proves highly effective for real-time applications. While the system outperforms leading commercial solutions, the UFPR-ALPR dataset highlights remaining challenges in unconstrained environments, such as highly tilted plates and low-light conditions. Future work will focus on plate rectification and layout-independent recognition to improve international applicability.

Summary updated

Abstract

Automatic License Plate Recognition (ALPR) has been a frequent topic of research due to many practical applications. However, many of the current solutions are still not robust in real-world situations, commonly depending on many constraints. This paper presents a robust and efficient ALPR system based on the state-of-the-art YOLO object detector. The Convolutional Neural Networks (CNNs) are trained and finetuned for each ALPR stage so that they are robust under different conditions (e.g., variations in camera, lighting, and background). Specially for character segmentation and recognition, we design a two-stage approach employing simple data augmentation tricks such as inverted License Plates (LPs) and flipped characters. The resulting ALPR approach achieved impressive results in two datasets. First, in the SSIG dataset, composed of 2,000 frames from 101 vehicle videos, our system achieved a recognition rate of 93.53% and 47 Frames Per Second (FPS), performing better than both Sighthound and OpenALPR commercial systems (89.80% and 93.03%, respectively) and considerably outperforming previous results (81.80%). Second, targeting a more realistic scenario, we introduce a larger public dataset <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> dataset, designed to ALPR. This dataset contains 150 videos and 4,500 frames captured when both camera and vehicles are moving and also contains different types of vehicles (cars, motorcycles, buses and trucks). In our proposed dataset, the trial versions of commercial systems achieved recognition rates below 70%. On the other hand, our system performed better, with recognition rate of 78.33% and 35 FPS.The UFPR-ALPR dataset is publicly available to the research community at https://web.inf.ufpr.br/vri/databases/ufpr-alpr/ subject to privacy restrictions.
BibTeX
@inproceedings{laroca2018robust,
  author = {Rayson Laroca and Evair Severo and Luiz A. Zanlorensi and Luiz S. Oliveira and Gabriel Resende Gonçalves and William Robson Schwartz and David Menotti},
  title = {{A Robust Real-Time Automatic License Plate Recognition Based on the YOLO Detector}},
  booktitle = {IJCNN},
  year = {2018},
  doi = {10.1109/ijcnn.2018.8489629},
}
Figure