SeeTek: Very Large-Scale Open-Set Logo Recognition With Text-Aware Metric Learning

Chenge Li, István Fehérvári, Xiaonan Zhao, Ives Macedo, Srikar AppalarajuWACV 2022

TL;DR

This paper introduces SeeTek, a novel multi-task learning framework for open-set logo recognition that combines visual and textual information. By jointly training a deep metric learning model with a scene text recognition branch, SeeTek significantly improves logo recognition accuracy, especially for visually similar or text-heavy logos, and scales to thousands of brands without retraining.

What Problem Does This Paper Address?

Recognizing logos in images is crucial for applications like product search and brand monitoring. However, this task is challenging due to visually similar logos, variations in logo appearance, and the need to handle a vast and ever-growing number of brands (open-set recognition). Existing methods often struggle with distinguishing visually similar logos, especially when dealing with thousands of brands, and typically require retraining for new brands. Furthermore, many logos incorporate text, which can be a vital cue for differentiation, but integrating this textual information into existing logo recognition pipelines is complex and often requires separate models.

Methods

The core of SeeTek is a multi-task learning architecture that combines two branches:

  • Visual Embedding Branch: This branch extracts visual features from logo images using a standard deep learning architecture (e.g., ResNet50).
  • Scene Text Recognition Branch: This branch is designed to extract textual information from logos. It uses a scene text recognition model, pre-trained on synthetic data and then fine-tuned using brand names as weak labels.

These two branches are trained jointly using a multi-task loss function that includes a classification-based deep metric learning (DML) loss and a scene text recognition loss. The DML loss aims to learn a discriminative latent space where similar logos are close and dissimilar ones are far apart. The text recognition loss leverages brand names to guide the model in extracting textual features. The final retrieval index is formed by concatenating the visual and textual embeddings. This joint training allows the model to learn complementary representations, where visual features handle general logo appearance and textual features disambiguate visually similar logos.

Datasets

The authors introduce a new, large-scale dataset called PL8K, containing 3,017,146 logo crops from nearly 8,000 brands, collected from Amazon Product Catalog images. This dataset is used for training and in-domain evaluation.

In addition to PL8K, the paper evaluates SeeTek on several public datasets:

  • LogoDet-3K
  • FlickrLogos-47
  • Logos-in-the-Wild (LitW)
  • QMUL-OpenLogo
  • BelgaLogos

A "hard evaluation dataset" was also created, consisting of false positives from other models to stress-test the system's robustness.

Key Insights and Ideas

  • Text as a Disambiguation Cue: Visually similar logos are a major hurdle. The paper demonstrates that incorporating text recognition, even with noisy brand name labels, provides a powerful complementary signal that significantly improves discrimination.
  • Weak Supervision for Text: Directly annotating text within logos is labor-intensive. Using brand names as weak labels for text recognition is an effective strategy to leverage readily available information and improve model performance.
  • Joint Training Benefits Visual Features: The text branch doesn't just add textual information; it also positively impacts the visual branch. The visual embeddings from the SeeTek model (trained jointly) outperform visual embeddings from a model trained solely on visual data, suggesting that text supervision helps the visual branch focus on more discriminative regions.
  • Scalability of Open-Set Recognition: The DML approach, combined with the text-aware metric learning, naturally handles open-set recognition, meaning it can recognize brands not seen during training without requiring retraining.

Key Results

  • SeeTek significantly outperforms previous state-of-the-art methods on various logo recognition benchmarks, particularly on the challenging PL8K dataset and out-of-domain datasets.
  • On the LogoDet-3K dataset, SeeTek achieved a Mean Average Precision (mAP) of 70.46% in an end-to-end open-set setting, compared to 52.28% for a closed-set YOLOv3 approach.
  • In embedder-level evaluations, SeeTek consistently achieved higher recall@1 scores than visual-only models across multiple datasets. For instance, on the PL8K dataset, the visual-textual embedding achieved 97.86% recall@1, while the visual-only model achieved 91.20%.
  • The attention-based text prediction head generally outperformed the CTC-based head.

Limitations and Future Work

  • Noisy Labels: While brand names are used as weak labels for text recognition, they can be noisy (e.g., logos with stylized text or partial brand names). The paper acknowledges that the text recognition performance is not perfect and could be improved.
  • Language Specificity: The current text recognition branch is primarily trained on English text. While it can benefit any logo with English alphabets, extending it to handle multiple languages would be a valuable future direction.
  • Out-of-Domain Generalization: While SeeTek shows good generalization, further research could explore more robust methods for handling extreme domain shifts or novel logo types.
  • Text-Heavy Bias: The paper notes that the text branch works best for text-heavy logos. While it doesn't harm performance on non-text logos, further work could investigate how to better balance the contributions of visual and textual information for different logo types.

Summary updated

BibTeX
@inproceedings{li2022seetek,
  author = {Chenge Li and István Fehérvári and Xiaonan Zhao and Ives Macedo and Srikar Appalaraju},
  title = {{SeeTek: Very Large-Scale Open-Set Logo Recognition With Text-Aware Metric Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure