Abstract
Egocentric scene images, captured by wearable cameras, present significant challenges for text recognition due to motion blur, perspective distortion, and environmental clutter. Traditional word spotting methods often rely on closed dictionaries or computationally expensive bounding box proposals. This paper introduces the Soft-PHOC descriptor, a mid-level representation that maps character probabilities at the pixel level using Fully Convolutional Networks (FCNs). By employing a query-driven line detection approach and Dynamic Time Warping (DTW) scoring, the proposed method enables robust, dictionary-free word spotting in complex real-world environments.
Introduction
Reading text in natural scenes is a fundamental task for scene understanding, with critical applications in assistive technologies for the visually impaired. However, "text in the wild"—particularly in egocentric images—is notoriously difficult to process. Most existing frameworks suffer from two main limitations: a reliance on pre-defined dictionaries, which prevents the recognition of arbitrary strings like URLs or dates, and a dependency on exhaustive bounding box proposals that often fail under extreme distortion.
To address these issues, we propose the Soft-PHOC (Pyramidal Histogram of Characters) descriptor. This representation is both character-based and spatially aware, allowing for flexible and robust word spotting without the need for traditional object detection proposals.
Methodology
The Soft-PHOC Representation
While a standard PHOC represents a word as a binary vector, the Soft-PHOC is a spatial tensor of dimension $H \times W \times 38$. Each channel corresponds to a specific character class (alphanumeric plus punctuation) and functions as a heatmap indicating the probability of that character's presence at every pixel. Unlike strong character labeling, Soft-PHOC provides a smoother, probabilistic distribution that is more resilient to noise.
Learning with Fully Convolutional Networks
The system utilizes an FCN architecture, inspired by FCN-8s, to predict these probability maps. A significant challenge in this domain is the extreme class imbalance, as background pixels typically constitute over 90% of the image. To mitigate this, a joint loss strategy is employed using three distinct masks to focus the model's attention on text regions and their immediate context.
Query-Driven Line Detection and Scoring
Unlike traditional methods that search for boxes, our approach identifies text lines. Given a query word:
- Bigram Heatmaps: Character maps of adjacent letters are multiplied to identify likely sequences.
- Hough Transform: This technique is applied to the bigram maps to detect candidate text lines.
- DTW Scoring: The system traverses each candidate line, using Dynamic Time Warping to compare predicted Soft-PHOC values against the ideal representation of the query.
Experimental Evaluation
The model was pre-trained on synthetic data and then evaluated on the ICDAR 2015 Challenge 4 (Incidental Scene Text) dataset. This dataset is particularly challenging as it contains images captured incidentally by wearable cameras, featuring heavy clutter and motion blur.
Results
On the ICDAR 2015 benchmark using a strongly contextualized query list:
- Line Detection: Achieved a Precision of 0.58 and a Recall of 0.57 at an overlap threshold of 0.5.
- End-to-End Spotting: When converted to bounding boxes for comparison with state-of-the-art methods, the system reached an H-mean of 0.24.
While the box-based metrics are lower than some specialized pipelines, the Soft-PHOC approach excels in its query-driven nature, searching specifically for the target word rather than generating generic proposals.
Discussion and Conclusion
The shift from "detect then recognize" to a unified probabilistic mapping offers a more resilient path for computer vision in cluttered environments. By prioritizing lines over boxes, the system becomes more robust to perspective distortion. Future work will focus on handling unconventional text layouts and integrating language models to refine character sequence predictions. Soft-PHOC represents a significant step toward flexible, dictionary-free text spotting in the wild.
Summary updated