Optimal Transport-based Identity Matching for Identity-invariant Facial Expression Recognition

Dae Ha Kim, Byung Cheol SongNeurIPS 2022

Introduction: The Challenge of Inter-Identity Variation

Facial Expression Recognition (FER) is notoriously difficult because of inter-identity variation. A "smile" from a 20-year-old woman looks physically different from a "smile" by a 70-year-old man due to differences in bone structure, skin elasticity, gender, and culture.

Most AI models struggle to separate the emotion from the person. They often inadvertently learn identity-specific traits instead of pure emotional features. This is especially problematic in Valence-Arousal (VA) FER, where emotions are mapped onto a continuous 2D scale (measuring how positive/negative and how intense an emotion is). To address this, researchers introduce ELIM (Expression Learning with Identity Matching), a framework that uses Optimal Transport to decouple facial expressions from individual identities, achieving state-of-the-art, identity-invariant recognition.

Methods: The ELIM Framework

The core idea behind ELIM is to treat each person's identity as a unique "domain" and then normalize the features to remove domain-specific (identity-specific) noise.

The ELIM framework architecture showing identity matching via Sinkhorn iteration.

The framework operates through several key steps:

  1. Optimal Transport (OT) Matching: Within a training batch, the model picks a "reference" person and matches their expressions to "target" people. To find the most similar expressions across different IDs, it formulates the task as an Optimal Transport problem—finding the most efficient way to "move" features from one identity to another.
  2. Sinkhorn-Knopp Iteration: Because standard OT is computationally expensive, the authors utilize the Sinkhorn-Knopp algorithm. This fast, iterative method makes OT feasible for deep learning architectures.
  3. Gumbel-topk Sampling: Not every frame is useful (e.g., neutral faces provide little information for matching). The model uses a re-parameterized gradient estimator (Gumbel-topk) to sample only the most expressive features for the matching process.
  4. Identity Shift Normalization: Once expressions are matched, the model calculates an "ID shift vector" ($\mu$). This vector represents the identity-specific bias. By subtracting this shift and scaling the features, the model produces truly identity-invariant representations.

Key Insights and Datasets

The model was evaluated on three major "in-the-wild" datasets—AFEW-VA, Aff-wild, and Aff-wild2—which feature spontaneous behaviors and diverse demographics, making them much more challenging than controlled lab environments.

A clever insight of this work is treating "Identity" as a domain generalization problem. Just as a model might struggle between "daytime" and "nighttime" photos, FER models struggle between different people. By using OT, the model looks at the distribution of expressions across a whole batch rather than individual pairs, making the matching robust to outliers. Furthermore, weighting features based on the intensity of the emotion (the norm of the VA label) significantly improves matching accuracy.

Results and Performance

ELIM establishes new state-of-the-art performance, outperforming previous leading methods (like CAF) across almost all metrics. On the Aff-wild dataset, it improved the Concordance Correlation Coefficient (CCC)—a key metric for VA FER—by up to 10% or more.

Qualitative comparison of ELIM against baselines in the Valence-Arousal space for Angry, Sad, and Surprise expressions.

Despite using complex optimization, the Sinkhorn-Knopp approach proved to be roughly 3.7x faster than standard optimization tools (like QPTH) while maintaining nearly identical accuracy. Qualitatively, ELIM successfully clusters similar expressions from different people closer together in the feature space, proving it learns identity-invariant features. As seen in the VA space visualizations, ELIM's predictions consistently land closer to the Ground Truth (GT) than baseline models across various emotional states.

Limitations and Future Work

While highly effective, ELIM has a few limitations:

  • Reference ID Dependency: The model's learning in each batch currently depends on a single, randomly selected "reference" identity. Future work could explore dynamic or multi-reference connection methods.
  • Static vs. Temporal: ELIM operates on static images. Because facial expressions inherently unfold over time, incorporating temporal information from video sequences could further boost performance.
  • Ethical Concerns: The authors explicitly note the potential for misuse in unauthorized emotional monitoring and call for the development of privacy-aware FER frameworks.

Summary updated

BibTeX
@inproceedings{kim2022optimal,
  author = {Dae Ha Kim and Byung Cheol Song},
  title = {{Optimal Transport-based Identity Matching for Identity-invariant Facial Expression Recognition}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
}
Figure