TL;DR: The paper introduces DELVE (Deep sELf-expressiVE model), a novel "white-box" deep learning architecture for clustering data that lies on multiple nonlinear manifolds. By stacking self-expressive layers that progressively linearize the data, DELVE offers a highly interpretable, robust, and scalable alternative to traditional black-box deep subspace clustering methods.
Abstract
Self-expressive model is a method for clustering data drawn from a union of low-dimensional linear subspaces. It gains a lot of popularity due to its: 1) simplicity, based on the observation that each data point can be expressed as a linear combination of the other data points, 2) provable correctness under broad geometric and statistical conditions, and 3) many extensions for handling corrupted, imbalanced, and large-scale real data. This paper extends the self-expressive model to a Deep sELf-expressiVE model (DELVE) for handling the more challenging case that the data lie in a union of nonlinear manifolds. DELVE is constructed from stacking self-expressive layers, each of which maps each data point to a linear combination of the other data points, and can be trained via minimizing self-expressive losses.
With such a design, the operator, architecture, and training of DELVE have the explicit interpretation of producing progressively linearized representations from the input data in nonlinear manifolds. Moreover, by leveraging existing understanding and techniques for self-expressive models, DELVE has a collection of benefits such as design choice by principles, robustness via specialized layers, and efficiency via specialized optimizers. We demonstrate on image datasets that DELVE can effectively perform data clustering, remove data corruptions, and handle large scale data.
Introduction
High-dimensional data, such as images and video, often intrinsically lie on low-dimensional structures. When these structures are linear, traditional self-expressive models—which operate on the principle that each data point can be expressed as a linear combination of other points in the same subspace—perform exceptionally well for clustering. However, real-world data typically resides on nonlinear manifolds.
To address this, researchers often employ deep learning techniques like autoencoders to map data into a linear latent space. However, standard neural network layers (e.g., CNNs, MLPs, ReLUs) are often treated as "black boxes" that do not explicitly account for the data's intrinsic geometric structure. DELVE (Deep sELf-expressiVE model) addresses this by jointly learning a linearized representation and clustering the data using an architecture that is mathematically grounded and explicitly designed to model manifold structures.
The DELVE Architecture
The core contribution of this work is the Deep sELf-expressiVE model (DELVE). Unlike standard neural networks, DELVE is composed of stacked self-expressive layers. In a traditional layer, a learned weight matrix $W$ is applied to individual data points ($WX$). In contrast, a self-expressive layer computes a linear combination of all other data points in the dataset ($XC$), where $C$ is a learned coefficient matrix.
Progressive Linearization
Because points on a nonlinear manifold are only locally linear, a single self-expressive operation produces an output that is slightly more "flattened" than the input. By stacking these layers, DELVE progressively linearizes the underlying nonlinear manifolds into linear subspaces. This process is trained layer-by-layer in an unsupervised manner, where each layer minimizes a self-expressive loss using regularizers like the Elastic Net.
Architectural Connections
DELVE offers a "white-box" alternative to deep learning, allowing for principled design choices. The self-expressive operation ($XC$) shares a mathematical foundation with other modern architectures. As shown in the architectural comparison below, while Self-Attention computes $C$ via queries and keys, and Graph Convolutional Networks (GCNs) use a fixed adjacency matrix, DELVE learns $C$ directly from the data's geometric relationships to capture the manifold structure.
Experimental Results
DELVE was evaluated on several benchmarks, including COIL-100, Extended Yale B (EYaleB), and MNIST. The model achieved state-of-the-art or highly competitive clustering accuracy: 89.76% on EYaleB, 78.96% on COIL-100 (reaching 84.64% with a locality constraint), and 96.38% on MNIST.
Beyond accuracy, DELVE demonstrated remarkable robustness. On the EYaleB dataset, it maintained high clustering performance even when up to 70% of the pixels were corrupted with sparse noise. Furthermore, its layer-wise training approach is highly memory-efficient, enabling it to cluster the entire 70,000-sample MNIST dataset—a task that often causes competing deep subspace clustering networks to exceed memory limits.
Conclusion
DELVE provides an interpretable and robust framework for clustering data on nonlinear manifolds. By stacking self-expressive layers, it achieves progressive linearization with clear mathematical interpretations. While future work is needed to address inference on new data and provide stronger theoretical guarantees for nonlinear cases, DELVE represents a significant advancement in "white-box" deep learning for geometric data analysis.
Summary updated