Learning Deep Convolutional Embeddings for Face Representation Using Joint Sample- and Set-Based Supervision

Baris Gecer, Vassileios Balntas, Tae-Kyun Kim

Abstract

This work investigates strategies for learning deep embeddings for face recognition through joint sample- and set-based optimization. We introduce a framework that augments traditional sample-based learning with set-based supervision, utilizing specific strategies to maintain set characteristics. Central to this approach is a novel Max-Margin Loss, which leverages Support Vector Machines (SVMs) to maximize inter-class margins. This loss implicitly directs samples toward the correct side of a separating hyperplane with a force inversely proportional to their distance from it. Empirical results on the LFW and YouTube Faces (YTF) benchmarks demonstrate that this joint supervision outperforms traditional methods.

Introduction

Standard deep learning models for face recognition typically rely on "sample-based" losses, such as Softmax or Triplet Loss, which optimize representations based on individual images or small tuples. However, these methods often overlook the collective information present in a set of images belonging to the same identity—capturing variations in pose, lighting, and expression. While existing set-based methods like Center Loss focus on intra-class compactness, they often fail to explicitly optimize the inter-class boundaries. This paper addresses the challenge of defining and optimizing these margins within a standard neural network training pipeline.

The Joint Training Framework

The proposed architecture integrates two parallel supervision streams: a standard sample-based loss and a set-based loss.

Overview of the joint training framework showing sample-based and set-based supervision paths.

As shown in Figure 1, the framework utilizes two sampling strategies:

  1. Random-sampling: Feeds individual images for standard sample-based loss (e.g., Softmax).
  2. Set-sampling: Groups images by identity to calculate set-based statistics.

To maintain accurate set parameters (centroids $\mu$ and SVM margins $\omega, b$) as the network's feature space evolves, the authors employ two update mechanisms:

  • Offline Updates: Conducted every $n$ iterations to recalculate global statistics across a large data subset.
  • Online Updates: A running average approach that adjusts parameters incrementally at each training step.

Max-Margin Loss

The core innovation is the Max-Margin Loss ($L_M$). Unlike Center Loss, which pulls samples toward a centroid, $L_M$ focuses on the boundaries between classes.

Visualization of the Max-Margin Loss mechanism: (a) initial state with forces, (b) zoomed view, (c) state after one update, and (d) convergence.

The optimization process involves:

  1. SVM Integration: Periodically calculating optimal separating hyperplanes between identities using a Linear SVM.
  2. Exponential Pushing: The loss function applies a "pushing" force to samples. If a sample is on the wrong side of the hyperplane, the strength of the push grows exponentially, aggressively correcting the misclassification (see Figure 3).

Experimental Evaluation

The model was trained on a curated version of the VGG Face dataset (0.83M images, 2,558 identities) and evaluated on two major benchmarks:

  • Labeled Faces in the Wild (LFW): Achieved 96.03% accuracy.
  • YouTube Faces (YTF): Achieved 92.44% accuracy, significantly benefiting from the set-based approach which aligns well with video-based face recognition.

Conclusion and Future Work

Joint sample- and set-based supervision provides a data-efficient pathway to highly discriminative features. By incorporating geometric margin maximization via SVMs, the model learns to not just separate classes, but to maximize the distance between them. Future research will focus on reducing the computational overhead of offline SVM updates and refining the interaction between online parameter updates and stochastic gradient descent.

Summary updated

Abstract

In this work, we investigate several methods and strategies to learn deep embeddings for face recognition, using joint sample- and set-based optimization. We explain our framework that expands traditional learning with set-based supervision together with the strategies used to maintain set characteristics. We, then, briefly review the related set-based loss functions, and subsequently we propose a novel Max-Margin Loss which maximizes maximum possible inter-class margin with assistance of Support Vector Machines (SVMs). It implicitly pushes all the samples towards correct side of the margin with a vector perpendicular to the hyperplane and a strength inversely proportional to the distance to it. We show that the introduced loss outperform the previous sample-based and set-based ones in terms verification of faces on two commonly used benchmarks.
BibTeX
@inproceedings{gecer2017learning,
  author = {Baris Gecer and Vassileios Balntas and Tae-Kyun Kim},
  title = {{Learning Deep Convolutional Embeddings for Face Representation Using Joint Sample- and Set-Based Supervision}},
  booktitle = {Workshops @ ICCV},
  year = {2017},
  url = {https://openreview.net/forum?id=oPtzROEhM4},
}

Loading related content...

Figure