Learning unbiased features

Yujia Li, Kevin Swersky, Richard S. ZemelarXiv 2014 · arXiv

Introduction

A fundamental challenge in machine learning is the tendency of deep neural networks to overfit to "nuisance" variables—irrelevant factors such as lighting conditions, backgrounds, or domain-specific noise. When these models are deployed in new environments (the "domain shift" problem), their performance often degrades significantly. This paper proposes a unified framework to learn unbiased, invariant representations by incorporating Maximum Mean Discrepancy (MMD) as a versatile regularization tool.

The MMD Framework

The core objective is to force a neural network's hidden layers to produce feature distributions that are indistinguishable across different domains or conditions. The authors achieve this by adding an MMD penalty to the standard loss function.

MMD is a kernel-based statistical test that measures the distance between two distributions, $P$ and $Q$. By minimizing this distance in a high-dimensional Reproducing Kernel Hilbert Space (RKHS), the network is incentivized to find a feature space where the "bias" (e.g., the difference between "Books" and "Electronics" review styles) is minimized. The researchers utilize the kernel trick—specifically Gaussian and Linear kernels—to measure these distances efficiently without explicit mapping.

Experimental Evaluation

1. Lighting-Invariant Face Recognition

Using the Extended Yale B Face Dataset, the authors tested whether MMD could help a network ignore lighting variations while maintaining identity recognition. By applying a Gaussian MMD penalty, the model was forced to align the feature distributions of images taken under different illumination conditions.

Comparison of face feature distributions without (a) and with (b) MMD regularization. The MMD-regularized model shows tighter clustering by identity rather than lighting.

As shown in the visualization above, the baseline model (a) clusters heavily based on lighting "nuisances." In contrast, the MMD-enhanced model (b) achieves a more identity-centric representation. Quantitatively, this approach improved test accuracy from a baseline of 72% to 82%.

2. Domain Adaptation in Sentiment Analysis

The framework was evaluated on the Amazon Product Review dataset, which involves transferring sentiment classification knowledge across categories (e.g., from Electronics to Kitchen appliances). The "NN MMD" model consistently outperformed standard Neural Networks and Support Vector Machines. For instance, on the Electronics $\to$ Kitchen task, the MMD-regularized network achieved 85.2% accuracy, surpassing the 84.5% achieved by a Linear SVM.

3. Generative Modeling and Noise Invariance

The paper also explores MMD as a stable alternative to the adversarial objectives used in Generative Adversarial Networks (GANs). By directly matching the distribution of generated samples to real data, the model avoids the minimax instability common in adversarial training.

MNIST generative modeling results. (a) shows learned filters/reconstructions and (b) shows generated digit samples using the MMD framework.

Furthermore, the authors demonstrated that MMD-based autoencoders produce features significantly more robust to noise. In tests distinguishing clean from noisy data based on latent features, the MMD model showed the highest invariance, with an SVM only able to distinguish the two with 61.1% accuracy (compared to 82.5% for a standard Denoising Autoencoder).

Key Insights and Limitations

  • Distributional vs. Parameter Regularization: Unlike weight decay, MMD acts on the global statistics of representations, providing a more direct constraint on how the network perceives data groups.
  • Theoretical Unification: The authors demonstrate that the Contractive Auto-Encoder (CAE) penalty is essentially a specific instance of an MMD penalty with a linear kernel.
  • Complexity and Scaling: A primary limitation is the $O(N^2)$ computational complexity of MMD relative to sample size $N$. While effective for mini-batch training, scaling to massive datasets requires further optimization or approximation techniques.

Conclusion

By treating invariance as a distributional alignment problem, this work provides a robust mathematical framework for learning unbiased features. The success across domain adaptation, face recognition, and generative modeling suggests that MMD is a powerful, general-purpose tool for improving the transferability and stability of deep learning models.

Summary updated

Abstract

A key element in transfer learning is representation learning; if representations can be developed that expose the relevant factors underlying the data, then new tasks and domains can be learned readily based on mappings of these salient factors. We propose that an important aim for these representations are to be unbiased. Different forms of representation learning can be derived from alternative definitions of unwanted bias, e.g., bias to particular tasks, domains, or irrelevant underlying data dimensions. One very useful approach to estimating the amount of bias in a representation comes from maximum mean discrepancy (MMD) [5], a measure of distance between probability distributions. We are not the first to suggest that MMD can be a useful criterion in developing representations that apply across multiple domains or tasks [1]. However, in this paper we describe a number of novel applications of this criterion that we have devised, all based on the idea of developing unbiased representations. These formulations include: a standard domain adaptation framework; a method of learning invariant representations; an approach based on noise-insensitive autoencoders; and a novel form of generative model.
BibTeX
@misc{li2014learning,
  author = {Yujia Li and Kevin Swersky and Richard S. Zemel},
  title = {{Learning unbiased features}},
  year = {2014},
  eprint = {1412.5244},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1412.5244},
}
Figure