LRSC: Learning Representations for Subspace Clustering

Changsheng Li, Chen Yang, Bo Liu, Ye Yuan, Guoren Wang

Abstract

Subspace clustering is a fundamental unsupervised learning task that assumes high-dimensional data points are drawn from a union of low-dimensional subspaces. Traditional deep subspace clustering methods often suffer from data scarcity and architectural limitations due to the computational cost of self-expression layers. This paper introduces LRSC (Learning Representations for Subspace Clustering), a framework that leverages external data through meta-learning and knowledge distillation, alongside a novel self-supervised auxiliary task. By integrating task-level and model-level knowledge transfer, LRSC learns more discriminative representations, achieving state-of-the-art performance on various image and text benchmarks.

Introduction

Subspace clustering aims to partition data into groups such that each group corresponds to a specific linear subspace. While deep learning has significantly advanced this field by learning non-linear mappings, two primary challenges remain. First, most methods are "target-data-only," ignoring potentially valuable information from external labeled datasets. Second, the "self-expression layer"—a core component of deep subspace clustering—scales quadratically with the number of samples ($O(n^2)$), necessitating the use of relatively shallow networks to maintain computational feasibility. This "shallow bottleneck" prevents models from capturing the complex features available to deeper architectures.

Proposed Methodology: The LRSC Framework

LRSC addresses these challenges through a multi-faceted approach involving meta-learning, knowledge distillation, and self-supervision.

The LRSC architecture featuring Feature Extraction, Encoder, Decoder, and Classification modules, integrated with Meta-Learning and Distillation paths.

1. Meta-Learning Module (Task-level Transfer)

To combat data scarcity, LRSC employs a meta-learning strategy inspired by Model-Agnostic Meta-Learning (MAML). The model is pre-trained on a diverse set of auxiliary tasks derived from external "meta-data." This phase optimizes the encoder to extract features that are inherently well-suited for subspace structures, allowing for rapid adaptation to the specific characteristics of the target dataset.

2. Distillation Module (Model-level Transfer)

To overcome the architectural constraints imposed by the self-expression layer, the authors utilize knowledge distillation. A deep "teacher" network is trained on external data to capture high-level semantic features. A shallower "student" network (the clustering encoder) is then trained to minimize a distilled loss relative to the teacher's outputs. This allows the student to benefit from "deep" features without the associated computational overhead during the clustering phase.

3. Self-Supervised Module

LRSC introduces a self-supervised auxiliary task to further refine the learned representations. By treating the input to the self-expression layer ($Z$) and its reconstructed output ($ZC$) as a positive pair in a contrastive-like loss, the model is forced to ensure consistency and discriminativeness in the subspace representation. This effectively turns the self-expression mechanism into a source of internal supervision.

Experimental Results

The LRSC framework was evaluated on four benchmark datasets: Fashion-MNIST and notMNIST for image data, and BBC News and 20 Newsgroups for text data. Performance was measured using Accuracy (ACC), Normalized Mutual Information (NMI), and Adjusted Rand Index (ARI).

Comparison of LRSC against state-of-the-art subspace clustering methods across four benchmark datasets.

As shown in the results table, LRSC consistently outperforms existing state-of-the-art methods like DSCN, DASC, and DPSC. Notably, on the notMNIST dataset, LRSC achieved an ACC of 0.608, a significant improvement over the 0.566 achieved by previous deep methods. On the BBC text dataset, the model reached an impressive 0.929 ACC.

Ablation Study

Ablation experiments confirmed the necessity of each component. The combination of task-level transfer (meta-learning), model-level transfer (distillation), and self-supervision yielded the highest performance, demonstrating that these strategies are complementary and essential for achieving state-of-the-art results.

Conclusion and Future Work

LRSC represents a significant shift in subspace clustering by moving beyond the limitations of target-only data and shallow architectures. By effectively bridging the gap between supervised pre-training and unsupervised clustering, it sets a new standard for the field. Future research may explore automated methods for selecting relevant meta-data to avoid negative transfer and investigate more complex self-supervised tasks to further enhance representation learning.

Summary updated

Abstract

Deep learning based subspace clustering methods have attracted increasing attention in recent years, where a basic theme is to non-linearly map data into a latent space, and then uncover subspace structures based upon the data self-expressiveness property. However, almost all existing deep subspace clustering methods only rely on target domain data, and always resort to shallow neural networks for modeling data, leaving huge room to design more effective representation learning mechanisms tailored for subspace clustering. In this paper, we propose a novel subspace clustering framework through learning precise sample representations. In contrast to previous approaches, the proposed method aims to leverage external data through constructing lots of relevant tasks to guide the training of the encoder, motivated by the idea of meta-learning. Considering limited layer structures of current deep subspace clustering models, we intend to distill knowledge from a deeper network trained on the external data, and transfer it into the shallower model. To reach the above two goals, we propose a new loss function to realize them in a joint framework. Moreover, we propose to construct a new pretext task for self-supervised training of the model, such that the representation ability of the model can be further improved. Extensive experiments are performed on four publicly available datasets, and experimental results clearly demonstrate the efficacy of our method, compared to state-of-the-art methods.
BibTeX
@inproceedings{li2021lrsc,
  author = {Changsheng Li and Chen Yang and Bo Liu and Ye Yuan and Guoren Wang},
  title = {{LRSC: Learning Representations for Subspace Clustering}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i9.17014},
}
Figure