Discriminative Residual Analysis for Image Set Classification With Posture and Age Variations

Chuan-Xian Ren, You-Wei Luo, Xiao-Lin Xu, Dao-Qing Dai, Hong Yan

TL;DR

This paper introduces Discriminative Residual Analysis (DRA), a novel framework for classifying sets of images (like video frames) rather than single samples. By shifting the focus from raw appearance to the "residual space"—the differences between image groups—and employing a "nonfeasance" strategy to avoid sampling errors, the authors achieve state-of-the-art accuracy on datasets with extreme age and posture variations.

What Problem Does This Paper Address?

In standard image classification, you look at one picture and name the object. In image set classification, you are given a batch of images (e.g., several frames from a surveillance camera or a photo album) and must identify the subject.

The challenge? Real-world variations like aging and posture changes are continuous and subtle. Often, the "intra-class" variance (how much one person's face changes as they age from 10 to 80) is much larger than the "inter-class" variance (the difference between two different people of the same age). This leads to overlapping data distributions where traditional classifiers like SVMs struggle. The authors aim to find a way to extract "discriminant" information that remains stable despite these messy, gradual variations.

Methods

The core innovation of this paper is moving the analysis from the appearance space to the residual space.

  1. Residual Representation: Instead of comparing raw pixels, the model looks at the "residuals"—the error vectors left over after trying to represent a test set using a training set. The intuition is that these residuals strip away common noise and variations, leaving behind the task-specific features that actually distinguish one person from another.
  2. Distance of Interest (DOI): The authors define a metric called DOI, which is the ratio of "related distance" (how well a set matches its own class) to "unrelated distance" (how well it matches other classes).
  3. PE vs. TE Models: They propose two models:
  4. Partial-Error (PE): Focuses only on the residuals of the most likely correct and incorrect classes.
  5. Total-Error (TE): A more global approach that considers all residual vectors across all classes.
  6. Nonfeasance Strategy (NFS): Traditional methods often try to build "unrelated groups" by picking the samples closest to the test set. However, if the data is noisy, you might accidentally pick samples that should have been in the related group, causing the model to learn the wrong thing. NFS solves this by simply using all other available classes to form the unrelated group, avoiding the "selection error" entirely.
  7. Regularization: To handle the "small sample size" problem (where you have high-dimensional data but few images), they use eigenvalue regularization and matrix exponential transformations to ensure the math remains stable.

Datasets

The authors evaluated their method on five major benchmarks:

  • LFW (Labeled Faces in the Wild): A standard face recognition benchmark.
  • LAG (Large Age-Gap): Specifically designed to test recognition across massive age differences (0 to 80 years).
  • AgeDB: A recent "in-the-wild" age dataset with manually cleaned labels.
  • Caltech101 & Caltech256: Standard object recognition datasets used to prove the method works beyond just faces.

They utilized deep features extracted from VggFace2-ResNet-50 and SE-ResNeXt-50 as the input for their DRA framework.

Key Insights and Ideas

  • Residuals as a Filter: The "aha moment" here is that the error in a linear regression (the residual) isn't just "trash"—it actually contains the most discriminative information because it represents the parts of the image that cannot be explained away by common variations like lighting or posture.
  • The Power of "Doing Nothing" (NFS): In machine learning, we often try to be clever with sample selection. This paper shows that in high-variance scenarios, a "nonfeasance" (doing nothing/not selecting) strategy is more robust because it prevents the model from being "tricked" by outliers that look like the target class.
  • Efficiency: By formulating the problem as a generalized eigenvalue decomposition, the training becomes incredibly fast compared to iterative deep learning approaches.

Key Results

The DRA method consistently outperformed state-of-the-art (SOTA) competitors like DARG and PDL:

  • Accuracy: On the LFW dataset, DRA-PE-eig achieved 99.73% accuracy.
  • Age Robustness: On the challenging LAG dataset (Scheme 2), DRA achieved 87.55%, beating the next best competitor by over 10%.
  • Speed: On the AgeDB dataset, the training time for DRA-PE-eig was only 1.6 seconds, compared to 38.5 seconds for DARG, while maintaining higher accuracy.
  • Small Sample Performance: DRA showed significant resilience when the number of training images per person was very low, a scenario where most deep learning classifiers fail.

Limitations and Future Work

While DRA performs exceptionally well with deep features, its performance on raw pixel data (though better than others) is still relatively low, suggesting it relies on a good feature extractor to start with.

The authors suggest that future work could involve extending this residual analysis framework to zero-shot image classification, where the model must recognize categories it has never seen during training. Additionally, integrating the DRA loss directly into the end-to-end training of a Convolutional Neural Network (CNN) could potentially yield even higher gains.

Summary updated

Abstract

Image set recognition has been widely applied in many practical problems like real-time video retrieval and image caption tasks. Due to its superior performance, it has grown into a significant topic in recent years. However, images with complicated variations, e.g., postures and human ages, are difficult to address, as these variations are continuous and gradual with respect to image appearance. Consequently, the crucial point of image set recognition is to mine the intrinsic connection or structural information from the image batches with variations. In this work, a Discriminant Residual Analysis (DRA) method is proposed to improve the classification performance by discovering discriminant features in related and unrelated groups. Specifically, DRA attempts to obtain a powerful projection which casts the residual representations into a discriminant subspace. Such a projection subspace is expected to magnify the useful information of the input space as much as possible, then the relation between the training set and the test set described by the given metric or distance will be more precise in the discriminant subspace. We also propose a nonfeasance strategy by defining another approach to construct the unrelated groups, which help to reduce furthermore the cost of sampling errors. Two regularization approaches are used to deal with the probable small sample size problem. Extensive experiments are conducted on benchmark databases, and the results show superiority and efficiency of the new methods.
BibTeX
@article{ren2020discriminative,
  author = {Chuan-Xian Ren and You-Wei Luo and Xiao-Lin Xu and Dao-Qing Dai and Hong Yan},
  title = {{Discriminative Residual Analysis for Image Set Classification With Posture and Age Variations}},
  journal = {IEEE Trans. Image Process.},
  year = {2020},
  url = {https://openreview.net/forum?id=M3n8ybCWYWV},
}

Loading related content...

Figure