Learning Feature Representations for Look-Alike Images

Ayça Takmaz, Thomas Probst, Danda Pani Paudel, Luc Van GoolWorkshops @ CVPR 2019 · OpenReview

Abstract

Human perception of visual similarity relies on information varying from low-level features such as texture and color, to high-level features such as objects and elements. While generic features learned for image or face recognition tasks somewhat correlate with the perceived visual similarity, they are found to be inadequate for matching look-alike images. In this paper, we learn the 'look-alike feature' embedding, capable of representing the perceived image similarity, by fusing low- and high-level features within a modified CNN encoder architecture. This encoder is trained using the triplet loss paradigm on look-alike image pairs. Our findings demonstrate that combining features from different layers across the network is beneficial for look-alike image matching, and clearly outperforms the standard pretrained networks followed by finetuning. Furthermore, we show that the learned similarities are meaningful, and capture color, shape, facial or holistic appearance patterns, depending upon context and image modalities.

Introduction

When humans describe two distinct entities as "looking alike," they are often responding to shared visual cues that transcend simple identity. This might include a shared color palette, a similar silhouette, or a specific facial expression, even when the subjects are entirely different—such as a celebrity and a piece of fruit. Standard computer vision models, particularly those optimized for face recognition like FaceNet, are designed to ignore these "distractions" to focus on identity-invariant features. Consequently, they often fail to capture the low-level and mid-level visual information that humans use for look-alike comparisons.

Human look-alikes: Luke Evans and Orlando Bloom share similar facial features and hairstyles.

Methodology

To bridge this gap, the researchers developed a hierarchical feature integration method based on the Inception-ResNet-v1 architecture. The key innovation lies in extracting and fusing features from multiple levels of the network hierarchy:

  1. Multi-Layer Extraction: Ten intermediate layers are selected from across the network.
  2. Dimension Reduction: Each layer's output undergoes $1 \times 1$ convolution and average pooling to create compact vectors.
  3. Feature Stacking: These vectors are stacked with the final FaceNet embedding, capturing a range from raw textures to complex semantic concepts.
  4. Learnable Fusion: A final $1 \times 1$ convolutional layer acts as a weighting mechanism to determine the importance of different layers for matching.
  5. Triplet Loss Training: The model is trained to minimize the distance between "look-alike" pairs (anchor and positive) relative to unrelated images (anchor and negative).

Experimental Setup and Datasets

The study primarily utilizes the Totally-Looks-Like (TLL) Dataset, which contains 6,016 pairs of images curated from the wild. Unlike standard face datasets, TLL includes a diverse mix of humans, animals, cartoons, and inanimate objects. The authors used 5,776 pairs for training and 240 pairs for testing.

Key Results and Insights

The proposed model demonstrated substantial improvements in retrieval tasks. For a gallery size of $m=10$, the model achieved 64% Top-1 accuracy, significantly outperforming vanilla FaceNet (45%) and the original TLL baseline (38%). Even as the gallery size increased to $m=60$, the model maintained a Top-1 accuracy of 39%, nearly double that of standard FaceNet (24%).

A bumper car with features resembling a face, demonstrating the model's ability to capture holistic shape and appearance patterns.

Qualitatively, the model successfully identifies similarities across different modalities. A critical finding was that freezing the pre-trained backbone and training only the integration layers yielded the best results, suggesting that standard networks already "see" the necessary features but require a specific fusion strategy to prioritize them for look-alike matching.

Conclusion

This research highlights that "look-alikeness" is not solely a high-level semantic property but relies on integrating features from early and middle layers of a neural network. By fusing these multi-scale representations, the model more closely mimics human perception of visual similarity. Future work will focus on localizing specific image regions that contribute most to the look-alike effect and expanding the variety of non-human subjects in the training data.

Summary updated

Abstract

Human perception of visual similarity relies on information varying from low-level features such as texture and color, to high-level features such as objects and elements. While generic features learned for image or face recognition tasks somewhat correlate with the perceived visual similarity, they are found to be inadequate for matching look-alike images. In this paper, we learn the 'look-alike feature' embedding, capable of representing the perceived image similarity, by fusing low- and high-level features within a modified CNN encoder architecture. This encoder is trained using the triplet loss paradigm on look-alike image pairs. Our findings demonstrate that combining features from different layers across the network is beneficial for look-alike image matching, and clearly outperforms the standard pretrained networks followed by finetuning. Furthermore, we show that the learned similarities are meaningful, and capture color, shape, facial or holistic appearance patterns, depending upon context and image modalities.
BibTeX
@inproceedings{takmaz2019learning,
  author = {Ayça Takmaz and Thomas Probst and Danda Pani Paudel and Luc Van Gool},
  title = {{Learning Feature Representations for Look-Alike Images}},
  booktitle = {Workshops @ CVPR},
  year = {2019},
  url = {https://openreview.net/forum?id=BBL8zBuQvt},
}
Figure