Transferable Deep Features for Keyword Spotting

George Retsinas, Giorgos Sfikas, Basilis Gatos

TL;DR

This paper explores using "deep features" (activations from hidden layers of a neural network) for keyword spotting (KWS), a task similar to image retrieval for words. They demonstrate that these deep features are more transferable and robust than standard network outputs, leading to state-of-the-art performance, especially when training and testing data come from different document collections.

What Problem Does This Paper Address?

Keyword spotting (KWS) is the challenge of finding specific words within document images. This is often framed as an image retrieval problem: given a query word image, find similar word images in a collection. While deep learning has shown great promise in KWS, most methods train a neural network and then use its final output layer as a descriptor for word images. However, this approach can lead to models that overfit to the training data, performing poorly on unseen document styles or collections. This paper investigates whether features extracted from earlier, hidden layers of a deep neural network (dubbed "deep features") can offer better generalization and transferability across different document types.

Methods

The authors leverage PHOCnet, a state-of-the-art deep convolutional neural network previously developed for KWS. Instead of solely using the network's final output, they extract features from its hidden layers (specifically, the outputs of the spp5, relu6, and relu7 layers). To further enhance these features, they explore two key techniques:

  • Manifold Learning (t-SNE): They apply t-Distributed Stochastic Neighbor Embedding (t-SNE) to reduce the dimensionality of the extracted deep features. The intuition is that high-dimensional data often lies on lower-dimensional manifolds, and t-SNE aims to preserve local neighborhood structures in this reduced space.
  • Dissimilarity Measures: They compare different ways to measure the similarity (or dissimilarity) between feature vectors. The standard Euclidean distance (L2) is compared against the Bray-Curtis dissimilarity (BC) and a normalized Euclidean distance (L2-normal), which is equivalent to cosine similarity.

Datasets

The experiments were conducted using several well-known handwritten document collections:

  • GW20: A collection with limited style variability.
  • IAM: A more challenging and diverse dataset with contributions from many writers.
  • Bentham14 and Modern14: Datasets introduced in the context of the ICFHR 2014 HKWS competition.

The models were trained on subsets of these datasets (e.g., IAM or GW20) and tested on various combinations, including cross-dataset evaluations.

Key Insights and Ideas

  • Deep Features are More Transferable: The most significant finding is that deep features (from hidden layers) are considerably more robust and transferable than the standard network output. This means they perform better when applied to document collections different from the one used for training, mitigating the overfitting problem.
  • Hidden Layers Outperform Output Layer: Across most experimental setups, features extracted from hidden layers consistently yielded better Keyword Spotting performance (measured by Mean Average Precision - MAP) compared to using the network's final output.
  • Manifold Learning Boosts Performance: Applying t-SNE for dimensionality reduction generally led to a noticeable improvement in KWS performance, suggesting that capturing the underlying manifold structure of the deep features is beneficial.
  • Normalization Matters: The L2-normal (cosine) distance and Bray-Curtis dissimilarity generally outperformed the raw Euclidean distance, highlighting the importance of considering feature magnitudes or relative proportions.
  • Overfitting is a Real Problem: The paper strongly emphasizes that models trained on limited datasets (like GW20) can exhibit severe overfitting, achieving near-perfect scores on their training data but failing dramatically on unseen data. Deep features help to alleviate this.

Key Results

  • Deep features consistently outperformed the network output, with performance gains often in the range of 5-10% MAP. In some cases, the difference was over 17% (e.g., training on IAM and testing on Modern14).
  • When training on GW20 and testing on IAM or Modern14, the network output yielded almost zero MAP, while deep features still provided respectable performance.
  • The proposed deep features, when trained on IAM and tested on Bentham14 and Modern14, achieved significantly superior results compared to other state-of-the-art KWS methods, even outperforming methods that used the same training data. For instance, on Modern14, their deep features achieved 91.1% MAP, compared to 74.0% for the PHOCNet output and much lower scores for other methods.

Limitations and Future Work

  • Layer Choice: While spp5, relu6, and relu7 were explored, the paper notes that extracting useful deep features from layers before the SPP layer is challenging due to variable dimensionality, a problem they did not explore further.
  • Manifold Learning Parameterization: The paper mentions that t-SNE's perplexity parameter influences results, but a deep dive into optimal parameter tuning for all scenarios was not the focus.
  • Dissimilarity Measure Nuances: While BC and L2-normal performed well, the paper acknowledges that BC is not a strict distance metric. Further exploration into the theoretical underpinnings of why these measures work well for deep features could be beneficial.

The authors' work strongly suggests that future KWS research should focus more on transferable features and cross-dataset evaluation, rather than solely optimizing performance on a single, potentially overfitted, dataset.

Summary updated

Abstract

Deep features, defined as the activations of hidden layers of a neural network, have given promising results applied to various vision tasks. In this paper, we explore the usefulness and transferability of deep features, applied in the context of the problem of keyword spotting (KWS). We use a state-of-the-art deep convolutional network to extract deep features. The optimal parameters concerning their application are subsequently studied: the impact of the choice of hidden layer, the impact of applying dimensionality reduction with a manifold learning technique, as well as the choice of dissimilarity measure used to retrieve relevant word images. Extensive numerical results show that deep features lead to state-of-the-art KWS performance, even when the test and training set come from different document collections.
BibTeX
@inproceedings{retsinas2017transferable,
  author = {George Retsinas and Giorgos Sfikas and Basilis Gatos},
  title = {{Transferable Deep Features for Keyword Spotting}},
  booktitle = {IWCIM@EUSIPCO},
  year = {2017},
  doi = {10.3390/proceedings2020089},
}
Figure