Abstract
Image set recognition has been widely applied in many practical problems like real-time video retrieval and image caption tasks. Due to its superior performance, it has grown into a significant topic in recent years. However, images with complicated variations, e.g., postures and human ages, are difficult to address, as these variations are continuous and gradual with respect to image appearance. Consequently, the crucial point of image set recognition is to mine the intrinsic connection or structural information from the image batches with variations. In this work, a Discriminant Residual Analysis (DRA) method is proposed to improve the classification performance by discovering discriminant features in related and unrelated groups. Specifically, DRA attempts to obtain a powerful projection which casts the residual representations into a discriminant subspace. Such a projection subspace is expected to magnify the useful information of the input space as much as possible, then the relation between the training set and the test set described by the given metric or distance will be more precise in the discriminant subspace. We also propose a nonfeasance strategy by defining another approach to construct the unrelated groups, which help to reduce furthermore the cost of sampling errors. Two regularization approaches are used to deal with the probable small sample size problem. Extensive experiments are conducted on benchmark databases, and the results show superiority and efficiency of the new methods.
Introduction
In standard image classification, models typically evaluate single samples. However, many real-world applications—such as surveillance video analysis or photo album organization—provide a batch of images, or an "image set," for identification.
The primary challenge in image set classification lies in handling continuous variations like aging and posture changes. Often, the "intra-class" variance (e.g., the same person at ages 10 and 80) exceeds the "inter-class" variance (e.g., the difference between two different people of the same age), leading to overlapping data distributions where traditional classifiers struggle. The goal of Discriminative Residual Analysis (DRA) is to find a way to extract discriminant information that remains stable despite these messy, gradual variations.
Methodology: Discriminative Residual Analysis (DRA)
The core innovation of this research is shifting analysis from the appearance space to the residual space. Instead of comparing raw pixels, the model analyzes "residuals"—the error vectors remaining after attempting to represent a test set using a training set. This process strips away common noise and variations, highlighting the task-specific features that distinguish one identity from another.
Distance of Interest (DOI) and Model Variants
The authors define a metric called the Distance of Interest (DOI), which is the ratio of "related distance" (how well a set matches its own class) to "unrelated distance" (how well it matches other classes). Based on this, two models are proposed:
- Partial-Error (PE): Focuses on the residuals of the most likely correct and incorrect classes.
- Total-Error (TE): A global approach considering all residual vectors across all classes.
Nonfeasance Strategy (NFS)
Traditional methods often build "unrelated groups" by selecting samples closest to the test set. In noisy data, this can lead to "selection errors" where samples from the target class are accidentally included in the unrelated group. The proposed Nonfeasance Strategy (NFS) avoids this by using all other available classes to form the unrelated group, ensuring a more robust learning process and preventing the model from being "tricked" by outliers.
Handling Small Sample Sizes
To ensure mathematical stability when dealing with high-dimensional data and limited images (the "small sample size" problem), the framework employs eigenvalue regularization and matrix exponential transformations.
Experimental Results
The DRA method was evaluated on five major benchmarks, including LFW (Labeled Faces in the Wild), LAG (Large Age-Gap), and AgeDB. Deep features from VggFace2-ResNet-50 and SE-ResNeXt-50 served as inputs.
- Accuracy: DRA-PE-eig achieved a near-perfect 99.73% on LFW. On the challenging LAG dataset, it reached 87.55%, outperforming competitors by over 10%.
- Efficiency: Training on the AgeDB dataset took only 1.6 seconds for DRA-PE-eig, compared to 38.5 seconds for previous state-of-the-art methods like DARG.
- Robustness: The method showed significant resilience in scenarios where the number of training images per person was very low, a situation where most deep learning classifiers fail.
Conclusion
By treating residuals as a filter for discriminative information and adopting a robust "nonfeasance" approach to sample selection, DRA provides a fast and accurate framework for image set classification. While the method relies on a good feature extractor, its efficiency and robustness to age and posture variations mark a significant step forward. Future work may involve extending this to zero-shot classification or integrating the DRA loss directly into end-to-end CNN training.
Summary updated