Consistent Representation Learning for High Dimensional Data Analysis

Stan Z. Li, Lirong Wu, Zelin ZangarXiv 2020 · arXiv

Introduction

In the field of high-dimensional data analysis, researchers typically navigate three primary tasks: dimensionality reduction, clustering, and visualization. Traditionally, these tasks are executed as a disjointed pipeline—for instance, utilizing an Autoencoder for feature extraction, followed by K-Means for clustering, and finally t-SNE or UMAP for visualization.

However, this "detached" workflow often results in representational inconsistency. Because each algorithm optimizes for a different objective, the underlying geometry of the data can shift between stages. A distinct cluster identified in the latent feature space may overlap or disappear entirely when projected into a 2D visualization, leading to misleading interpretations. Consistent Representation Learning (CRL) addresses this by unifying these tasks into a single, end-to-end neural network framework.

Methodology: The CRL Framework

CRL is built upon a multi-stage neural network architecture designed to maintain geometric integrity across different representational spaces.

Two-Stage Nonlinear Dimensionality Reduction (NLDR)

The network consists of two cascaded NLDR modules. The first module transforms high-dimensional input data into a latent feature space ($Z = X^{(m)}$), specifically optimized for clustering. The second module further reduces this latent space into a 2D or 3D visualization space ($V = X^{(L)}$).

Local Geometry-Preserving (LGP) Constraints

The core innovation of CRL is the application of Local Geometry-Preserving (LGP) constraints across the network layers. These constraints ensure that the local neighborhood structures of the original data are preserved throughout the transformation process.

Figure 1: The CRL architecture featuring cascaded NLDR modules and LGP constraints between the input (Head), latent (Mid), and visualization (Tail) layers.

As shown in Figure 1, LGP constraints are applied in three ways:

  1. Head-Mid: Preserves geometry from input to latent space.
  2. Mid-Tail: Preserves geometry from latent to visualization space.
  3. Head-Tail: A direct constraint from input to visualization space, which ablation studies show is critical for preventing noise in final plots.

These constraints utilize a fuzzy divergence loss to ensure that points proximal in the high-dimensional input remain proximal in both the clustering and visualization embeddings.

Clustering and Metrics

CRL employs the Louvain algorithm on the latent space, offering greater robustness for non-convex cluster shapes compared to traditional K-Means. To quantify performance, the authors introduced the Clustering-Visualization Inconsistency (CVI) metric, which measures the spatial distance in the 2D/3D embedding between misclustered points and their true cluster centers.

Experimental Results

The authors evaluated CRL against standard benchmarks including MNIST, USPS, Fashion-MNIST, and Coil-100.

Superior Accuracy and Consistency

CRL consistently outperformed traditional pipelines (such as AE+t-SNE or AE+UMAP) in both Clustering Accuracy (ACC) and Normalized Mutual Information (NMI). On the complex Coil-100 dataset, CRL achieved an ACC of 0.949, significantly higher than the 0.847 achieved by the next best method, N2D.

Figure 3: Visual comparison of CRL against raw data, t-SNE, and UMAP across multiple datasets. Black boxes highlight regions where traditional methods fail to maintain cluster separation that CRL preserves.

As illustrated in Figure 3, CRL produces visualizations that are more "honest" representations of the underlying data. While t-SNE and UMAP often show overlapping clusters (indicated by the black boxes), CRL maintains clear separation, resulting in significantly lower CVI scores.

Generalization

Unlike non-parametric methods like standard t-SNE, CRL is a trained parametric neural network. This allows it to generalize to unseen data points instantly without re-running the entire optimization, making it highly suitable for real-world streaming data applications.

Conclusion and Future Directions

CRL demonstrates that an end-to-end approach with explicit geometric constraints is essential for consistent high-dimensional data analysis. While currently implemented with Multi-Layer Perceptrons, future iterations could integrate Convolutional Neural Networks (CNNs) for more complex image datasets or incorporate self-supervised contrastive learning to further enhance feature representation without the need for labels.

Summary updated

Abstract

High dimensional data analysis for exploration and discovery includes three fundamental tasks: dimensionality reduction, clustering, and visualization. When the three associated tasks are done separately, as is often the case thus far, inconsistencies can occur among the tasks in terms of data geometry and others. This can lead to confusing or misleading data interpretation. In this paper, we propose a novel neural network-based method, called Consistent Representation Learning (CRL), to accomplish the three associated tasks end-to-end and improve the consistencies. The CRL network consists of two nonlinear dimensionality reduction (NLDR) transformations: (1) one from the input data space to the latent feature space for clustering, and (2) the other from the clustering space to the final 2D or 3D space for visualization. Importantly, the two NLDR transformations are performed to best satisfy local geometry preserving (LGP) constraints across the spaces or network layers, to improve data consistencies along with the processing flow. Also, we propose a novel metric, clustering-visualization inconsistency (CVI), for evaluating the inconsistencies. Extensive comparative results show that the proposed CRL neural network method outperforms the popular t-SNE and UMAP-based and other contemporary clustering and visualization algorithms in terms of evaluation metrics and visualization.
BibTeX
@misc{li2020consistent,
  author = {Stan Z. Li and Lirong Wu and Zelin Zang},
  title = {{Consistent Representation Learning for High Dimensional Data Analysis}},
  year = {2020},
  eprint = {2012.00481},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.00481},
}
Figure