Scaling Digital Screen Reading With One-Shot Learning and Re-Identification

James Charles, Stefano Bucciarelli, Roberto CipollaWACV 2021

TL;DR

This paper presents a novel mobile app that can accurately read digital displays on various devices using just a single example image per device type. By combining one-shot learning for screen detection and a deep re-identification network, the system automatically selects the correct reading module, enabling a user to scan hundreds of different meters without manual configuration.

What Problem Does This Paper Address?

Many essential digital devices, such as health monitors (blood pressure cuffs, glucose meters) and industrial gauges, still require manual data entry. This process is time-consuming, error-prone, and hinders efficient data analysis. While newer "smart" devices can wirelessly transmit data, upgrading existing equipment is costly, and Bluetooth pairing can be cumbersome. The authors aim to bridge this gap by creating a vision-based system that allows a mobile phone to reliably recognize and read digital meter screens, effectively retrofitting them with "smart" capabilities without hardware modifications.

Methods

The core of the system comprises three main components, each implemented as a deep convolutional neural network (CNN):

  • Meter Re-identification: This module creates a scale, translation, and rotation-invariant embedding of a target meter. This embedding is used to quickly search and retrieve the correct meter type from a gallery of known meters, enabling automatic selection of the appropriate screen reading module.
  • Meter-Specific Screen Detector: This network precisely locates and orients the digital meter's screen by recovering its four corner coordinates. A key innovation here is the use of a "modality converter" that bridges the domain gap between synthetic training data and real-world images. This converter can be adapted to various CNN backbones (e.g., MobileNet, MaskRCNN) and is particularly effective when processing edge-based features.
  • Digit Recognizer (LeDigit): This network takes the rectified screen image and extracts strings of characters, classifying them by type (e.g., date, time, numerical values). It's trained on synthetic data and uses a heatmap regression approach.

A crucial aspect of the screen detector training is one-shot learning. Instead of requiring extensive labeled data, the system uses a single hand-labeled template image of a meter. This template is then used to synthesize a large amount of training data by applying various transformations, lighting conditions, and backgrounds. The modality converter is essential for making this synthetic data generalize well to real-world images.

Datasets

The paper introduces two datasets for evaluation:

  • Screen-detection dataset: Consists of 30 images per digital meter (blood glucose monitor, kitchen scale, bathroom scale, digital multimeter, digital body thermometer). Images were captured with a mobile phone and feature varying backgrounds, lighting, rotations, scales, and occlusions. Screen corner coordinates were manually labeled.
  • Digit-recognition dataset: Rectified screen images from the five digital meters were semi-automatically recorded. Snapshots were taken under diverse conditions and manually labeled as "valid" (if a reading was present and all digits visible) or "invalid." For the blood glucose meter, multiple tags (glucose, time, date) were provided, while other meters had a single tag.

Key Insights and Ideas

  • One-Shot Learning for Screen Detection: The ability to train a robust screen detector with just a single labeled image is a significant advancement, drastically reducing the annotation burden.
  • Modality Converter for Domain Adaptation: The modality converter, especially when focusing on edge features, effectively bridges the gap between synthetic and real-world data, leading to improved generalization for screen detection.
  • Re-identification for Scalability: Using a re-identification system to automatically select the correct meter reading module is key to scaling the system to hundreds of different meter types without user intervention.
  • Edge Modality Dominance: The paper consistently shows that using edge-based features (via the modality converter) yields superior screen detection performance compared to RGB or grayscale inputs, particularly in challenging scenarios with reflections.
  • Efficiency on Mobile Devices: The system is designed to run in real-time on mobile phones, achieving high frame rates (over 50fps on an iPhone 11), making it practical for everyday use.

Key Results

  • Read Accuracy: The system achieves very high read accuracy, close to 100% for validated readings across all meters. Validation sensitivity and specificity are also near 100%.
  • Screen Detection Performance: The MobileNet-Edge model achieved 93% vertex accuracy (at a 15px threshold) and an average vertex difference of only 9px, outperforming several baselines, including MaskRCNN and a keypoint matching system.
  • Re-identification Accuracy: The re-identification system achieved 96% top-1 retrieval accuracy on a gallery of 10,000 digital meter images combined with their own dataset.
  • Computational Performance: Per-frame processing times are very low: 0.6ms for digit recognition and 4ms for screen detection on an iPhone 11, enabling approximately 50fps processing.

Limitations and Future Work

  • Occlusion: The paper notes that most failures in re-identification are due to strong occlusions of the meter by objects like hands.
  • Fixed Stage Training: While exploring fixed stage training for the modality converter, the results were consistently worse than other modalities, suggesting that early feature extraction might overfit.
  • Future Work: The authors mention the potential for rapid training of many meter readers and the commercial application of their framework for diabetes management. Further exploration into improving robustness to occlusions and potentially extending the system to other types of displays could be future directions.

Summary updated

BibTeX
@inproceedings{charles2021scaling,
  author = {James Charles and Stefano Bucciarelli and Roberto Cipolla},
  title = {{Scaling Digital Screen Reading With One-Shot Learning and Re-Identification}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure