Abstract
Subspace clustering is a fundamental unsupervised learning task that assumes high-dimensional data points are drawn from a union of low-dimensional subspaces. Traditional deep subspace clustering methods often suffer from data scarcity and architectural limitations due to the computational cost of self-expression layers. This paper introduces LRSC (Learning Representations for Subspace Clustering), a framework that leverages external data through meta-learning and knowledge distillation, alongside a novel self-supervised auxiliary task. By integrating task-level and model-level knowledge transfer, LRSC learns more discriminative representations, achieving state-of-the-art performance on various image and text benchmarks.
Introduction
Subspace clustering aims to partition data into groups such that each group corresponds to a specific linear subspace. While deep learning has significantly advanced this field by learning non-linear mappings, two primary challenges remain. First, most methods are "target-data-only," ignoring potentially valuable information from external labeled datasets. Second, the "self-expression layer"—a core component of deep subspace clustering—scales quadratically with the number of samples ($O(n^2)$), necessitating the use of relatively shallow networks to maintain computational feasibility. This "shallow bottleneck" prevents models from capturing the complex features available to deeper architectures.
Proposed Methodology: The LRSC Framework
LRSC addresses these challenges through a multi-faceted approach involving meta-learning, knowledge distillation, and self-supervision.
1. Meta-Learning Module (Task-level Transfer)
To combat data scarcity, LRSC employs a meta-learning strategy inspired by Model-Agnostic Meta-Learning (MAML). The model is pre-trained on a diverse set of auxiliary tasks derived from external "meta-data." This phase optimizes the encoder to extract features that are inherently well-suited for subspace structures, allowing for rapid adaptation to the specific characteristics of the target dataset.
2. Distillation Module (Model-level Transfer)
To overcome the architectural constraints imposed by the self-expression layer, the authors utilize knowledge distillation. A deep "teacher" network is trained on external data to capture high-level semantic features. A shallower "student" network (the clustering encoder) is then trained to minimize a distilled loss relative to the teacher's outputs. This allows the student to benefit from "deep" features without the associated computational overhead during the clustering phase.
3. Self-Supervised Module
LRSC introduces a self-supervised auxiliary task to further refine the learned representations. By treating the input to the self-expression layer ($Z$) and its reconstructed output ($ZC$) as a positive pair in a contrastive-like loss, the model is forced to ensure consistency and discriminativeness in the subspace representation. This effectively turns the self-expression mechanism into a source of internal supervision.
Experimental Results
The LRSC framework was evaluated on four benchmark datasets: Fashion-MNIST and notMNIST for image data, and BBC News and 20 Newsgroups for text data. Performance was measured using Accuracy (ACC), Normalized Mutual Information (NMI), and Adjusted Rand Index (ARI).
As shown in the results table, LRSC consistently outperforms existing state-of-the-art methods like DSCN, DASC, and DPSC. Notably, on the notMNIST dataset, LRSC achieved an ACC of 0.608, a significant improvement over the 0.566 achieved by previous deep methods. On the BBC text dataset, the model reached an impressive 0.929 ACC.
Ablation Study
Ablation experiments confirmed the necessity of each component. The combination of task-level transfer (meta-learning), model-level transfer (distillation), and self-supervision yielded the highest performance, demonstrating that these strategies are complementary and essential for achieving state-of-the-art results.
Conclusion and Future Work
LRSC represents a significant shift in subspace clustering by moving beyond the limitations of target-only data and shallow architectures. By effectively bridging the gap between supervised pre-training and unsupervised clustering, it sets a new standard for the field. Future research may explore automated methods for selecting relevant meta-data to avoid negative transfer and investigate more complex self-supervised tasks to further enhance representation learning.
Summary updated