Soft-PHOC Descriptor for End-to-End Word Spotting in Egocentric Scene Images

Dena Bazazian, Dimosthenis Karatzas, Andrew D. BagdanovarXiv 2018 · arXiv

Abstract

Egocentric scene images, captured by wearable cameras, present significant challenges for text recognition due to motion blur, perspective distortion, and environmental clutter. Traditional word spotting methods often rely on closed dictionaries or computationally expensive bounding box proposals. This paper introduces the Soft-PHOC descriptor, a mid-level representation that maps character probabilities at the pixel level using Fully Convolutional Networks (FCNs). By employing a query-driven line detection approach and Dynamic Time Warping (DTW) scoring, the proposed method enables robust, dictionary-free word spotting in complex real-world environments.

Introduction

Reading text in natural scenes is a fundamental task for scene understanding, with critical applications in assistive technologies for the visually impaired. However, "text in the wild"—particularly in egocentric images—is notoriously difficult to process. Most existing frameworks suffer from two main limitations: a reliance on pre-defined dictionaries, which prevents the recognition of arbitrary strings like URLs or dates, and a dependency on exhaustive bounding box proposals that often fail under extreme distortion.

To address these issues, we propose the Soft-PHOC (Pyramidal Histogram of Characters) descriptor. This representation is both character-based and spatially aware, allowing for flexible and robust word spotting without the need for traditional object detection proposals.

Methodology

The Soft-PHOC Representation

While a standard PHOC represents a word as a binary vector, the Soft-PHOC is a spatial tensor of dimension $H \times W \times 38$. Each channel corresponds to a specific character class (alphanumeric plus punctuation) and functions as a heatmap indicating the probability of that character's presence at every pixel. Unlike strong character labeling, Soft-PHOC provides a smoother, probabilistic distribution that is more resilient to noise.

Comparison between Strong Character Labeling and the proposed Soft-PHOC Labeling.

Learning with Fully Convolutional Networks

The system utilizes an FCN architecture, inspired by FCN-8s, to predict these probability maps. A significant challenge in this domain is the extreme class imbalance, as background pixels typically constitute over 90% of the image. To mitigate this, a joint loss strategy is employed using three distinct masks to focus the model's attention on text regions and their immediate context.

The FCN architecture used for predicting Soft-PHOC maps and the multi-mask loss calculation.

Query-Driven Line Detection and Scoring

Unlike traditional methods that search for boxes, our approach identifies text lines. Given a query word:

  1. Bigram Heatmaps: Character maps of adjacent letters are multiplied to identify likely sequences.
  2. Hough Transform: This technique is applied to the bigram maps to detect candidate text lines.
  3. DTW Scoring: The system traverses each candidate line, using Dynamic Time Warping to compare predicted Soft-PHOC values against the ideal representation of the query.

Experimental Evaluation

The model was pre-trained on synthetic data and then evaluated on the ICDAR 2015 Challenge 4 (Incidental Scene Text) dataset. This dataset is particularly challenging as it contains images captured incidentally by wearable cameras, featuring heavy clutter and motion blur.

Results

On the ICDAR 2015 benchmark using a strongly contextualized query list:

  • Line Detection: Achieved a Precision of 0.58 and a Recall of 0.57 at an overlap threshold of 0.5.
  • End-to-End Spotting: When converted to bounding boxes for comparison with state-of-the-art methods, the system reached an H-mean of 0.24.

While the box-based metrics are lower than some specialized pipelines, the Soft-PHOC approach excels in its query-driven nature, searching specifically for the target word rather than generating generic proposals.

Discussion and Conclusion

The shift from "detect then recognize" to a unified probabilistic mapping offers a more resilient path for computer vision in cluttered environments. By prioritizing lines over boxes, the system becomes more robust to perspective distortion. Future work will focus on handling unconventional text layouts and integrating language models to refine character sequence predictions. Soft-PHOC represents a significant step toward flexible, dictionary-free text spotting in the wild.

Summary updated

Abstract

Word spotting in natural scene images has many applications in scene understanding and visual assistance. In this paper we propose a technique to create and exploit an intermediate representation of images based on text attributes which are character probability maps. Our representation extends the concept of the Pyramidal Histogram Of Characters (PHOC) by exploiting Fully Convolutional Networks to derive a pixel-wise mapping of the character distribution within candidate word regions. We call this representation the Soft-PHOC. Furthermore, we show how to use Soft-PHOC descriptors for word spotting tasks in egocentric camera streams through an efficient text line proposal algorithm. This is based on the Hough Transform over character attribute maps followed by scoring using Dynamic Time Warping (DTW). We evaluate our results on ICDAR 2015 Challenge 4 dataset of incidental scene text captured by an egocentric camera.
BibTeX
@misc{bazazian2018soft,
  author = {Dena Bazazian and Dimosthenis Karatzas and Andrew D. Bagdanov},
  title = {{Soft-PHOC Descriptor for End-to-End Word Spotting in Egocentric Scene Images}},
  year = {2018},
  eprint = {1809.00854},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1809.00854},
}
Figure