Discriminant Tensor Dictionary Learning with Neighbor Uncorrelation for Image Set Based Classification

Fei Wu, Xiao-Yuan Jing, Wangmeng Zuo, Ruiping Wang, Xiaoke Zhu

Introduction

Image Set Based Classification (ISC) has emerged as a critical task in computer vision, particularly for applications like face recognition from video sequences. Unlike single-image classification, ISC leverages a collection of images to identify an object, providing a richer set of data that captures variations in pose, lighting, and expression. However, modeling these sets effectively remains a challenge. Traditional methods often struggle with high-dimensional data and the loss of spatial information. This paper introduces Discriminant Tensor Dictionary Learning with Neighbor Uncorrelation (DTDLNU), a framework designed to preserve the inherent structure of image sets while maximizing the discriminative power of learned dictionaries.

The Vectorization Trap

A significant limitation in many existing ISC methods is what can be termed the "Vectorization Trap." Most algorithms flatten 2D images into 1D vectors before processing. This transformation destroys the spatial relationships between pixels—the very geometry that defines an object's appearance. Furthermore, when learning dictionaries (sets of representative templates) for classification, these methods often produce sub-dictionaries for different classes that are highly correlated, especially for visually similar categories. This redundancy leads to increased misclassification rates.

Proposed Method: DTDLNU

To overcome these hurdles, DTDLNU shifts the paradigm from vector-based to tensor-based modeling.

Comparison of traditional vectorization (a) versus the proposed tensor-based representation (b).

1. Tensor Representation

Instead of treating an image set as a collection of vectors, DTDLNU models it as a 3rd-order tensor $\mathcal{Y} \in \mathbb{R}^{d_W \times d_H \times d_N}$, where $d_W$, $d_H$, and $d_N$ represent width, height, and the number of images, respectively. This approach preserves the spatial structure of the data throughout the learning process.

2. Triple Dictionary Learning

The framework jointly learns three structured dictionaries corresponding to the three modes of the tensor:

  • Width-mode and Height-mode dictionaries: These capture the spatial patterns within the images.
  • Set-mode dictionary: This captures the temporal or sequential variations across the image set.

By decomposing the data into these three modes, the model exploits the intrinsic high-order structure more effectively than flat dictionary learning.

3. Neighbor-Uncorrelation (NU) Constraint

The "secret sauce" of DTDLNU is the Neighbor-Uncorrelation term. Most dictionary learning methods attempt to make all class dictionaries different. DTDLNU specifically targets "neighboring" classes—those that are most likely to be confused. By minimizing the correlation between these specific sub-dictionaries, the model enhances its ability to distinguish between the hardest cases.

Experimental Evaluation

The authors evaluated DTDLNU on three prominent datasets: YouTube Celebrities (YTC), COX Face Database, and YouTube Faces (YTF). These datasets are characterized by real-world challenges such as low resolution, noise, and significant variations in expression and pose.

Key Results

DTDLNU consistently outperformed state-of-the-art methods. On the YTC dataset, it achieved an accuracy of 80.16%, surpassing the next best method by nearly 2%. On the massive YTF dataset, it reached 85.16%, notably outperforming several deep learning-based approaches.

Correlation values of dictionaries across different datasets, demonstrating the superior discriminative power of DTDLNU.

The effectiveness of the Neighbor-Uncorrelation term is clearly visible in the reduced correlation values between class dictionaries compared to other leading methods like ISCRC and DFRV, as shown in the figure above. This lower correlation directly translates to higher classification accuracy by making the templates for different people more distinct.

Key Insights

  • Structure is Information: Preserving the 3D tensor form allows the model to achieve higher accuracy with fewer parameters, as it respects the natural geometry of the image data.
  • Local Decorrelation: Focusing on decorrelating neighboring (similar) classes is more computationally efficient and effective for resolving common classification errors than global decorrelation.
  • Mathematical Modeling vs. Deep Learning: DTDLNU's success against deep learning models suggests that for specific tasks like ISC, structured mathematical modeling can be more powerful than "black-box" neural networks, particularly when data is structured but limited.

Conclusion

DTDLNU represents a significant step forward in image set classification. By combining tensor-based representation with a targeted neighbor-uncorrelation constraint, it provides a robust and discriminative framework that excels in challenging real-world scenarios. Future work may explore extending this to higher-order tensors for color video or integrating these structured dictionaries into deep learning architectures.

Summary updated

Abstract

Image set based classification (ISC) has attracted lots of research interest in recent years. Several ISC methods have been developed, and dictionary learning technique based methods obtain state-of-the-art performance. However, existing ISC methods usually transform the image sample of a set into a vector for subsequent processing, which breaks the inherent spatial structure of image sample and the set. In this paper, we utilize tensor to model an image set with two spatial modes and one set mode, which can fully explore the intrinsic structure of image set. We propose a novel ISC approach, named discriminant tensor dictionary learning with neighbor uncorrelation (DTDLNU), which jointly learns two spatial dictionaries and one set dictionary. The spatial and set dictionaries are composed by set-specific sub-dictionaries corresponding to the class labels, such that the reconstruction error is discriminative. To obtain dictionaries with favorable discriminative power, DTDLNU designs a neighbor-uncorrelated discriminant tensor dictionary term, which minimizes the within-class scatter of the training sets in the projected tensor space and reduces tensor dictionary correlation among set-specific sub-dictionaries corresponding to neighbor sets from different classes. Experiments on three challenging datasets demonstrate the effectiveness of DTDLNU.
BibTeX
@inproceedings{wu2017discriminant,
  author = {Fei Wu and Xiao-Yuan Jing and Wangmeng Zuo and Ruiping Wang and Xiaoke Zhu},
  title = {{Discriminant Tensor Dictionary Learning with Neighbor Uncorrelation for Image Set Based Classification}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/428},
}
Figure