Revealing hidden patterns in deep neural network feature space continuum via manifold learning

Md Tauhidul Islam, Zixia Zhou, Hongyi Ren, Masoud Badiei Khuzani, Daniel S. Kapp, James Zou, Lü Tian, Joseph C. Liao, Lei XingNature Communications 2023 · DOI

Abstract

Deep neural networks (DNNs) extract thousands to millions of task-specific features during model training for inference and decision-making. While visualizing these features is critical for comprehending the learning process and improving the performance of the DNNs, existing visualization techniques work only for classification tasks. For regressions, the feature points lie on a high dimensional continuum having an inherently complex shape, making a meaningful visualization of the features intractable. Given that the majority of deep learning applications are regression-oriented, developing a conceptual framework and computational method to reliably visualize the regression features is of great significance. Here, we introduce a manifold discovery and analysis (MDA) method for DNN feature visualization, which involves learning the manifold topology associated with the output and target labels of a DNN. MDA leverages the acquired topological information to preserve the local geometry of the feature space manifold and provides insightful visualizations of the DNN features, highlighting the appropriateness, generalizability, and adversarial robustness of a DNN. The performance and advantages of the MDA approach compared to the existing methods are demonstrated in different deep learning applications.

Introduction

Deep learning is often criticized as a "black box." While tools like t-SNE and UMAP have successfully illuminated the internal workings of classification networks by showing how they cluster data into discrete categories, a significant gap remains for regression tasks. In regression—common in medical imaging, survival prediction, and signal reconstruction—data exists on a continuous, high-dimensional manifold rather than in discrete buckets. Standard visualization tools often fail in this context, producing uninformative "spurious clusters" or noise-like clouds that obscure whether a model has truly captured the underlying data physics or is merely overfitting. This makes interpreting the "latent space" of the most common AI models nearly impossible.

Methods: Manifold Discovery and Analysis (MDA)

The core innovation of the Manifold Discovery and Analysis (MDA) framework is supervised manifold learning designed specifically for regression feature spaces. Instead of looking at features in isolation, MDA uses the relationship between the network's predicted outputs and target labels to guide the visualization.

Overview of DNN feature space manifolds across layers (a) and the step-by-step MDA workflow (b).

The MDA workflow consists of four primary steps:

  1. Manifold Outlining: The algorithm computes distances between predicted labels to identify the "flow" and endpoints of the output manifold.
  2. Pseudo-labeling: Outputs are sorted into bins (pseudo-labels) based on their position along the manifold.
  3. Bayesian Projection: A supervised Bayesian approach projects high-dimensional features into a lower-dimensional space while strictly preserving geodesic distances (distance measured along the curved manifold surface).
  4. Deep Learning Embedding: A small neural network transforms these projections into a 2D visualization optimized to show the continuous transition of data points.

Results and Discussion

MDA was validated across diverse datasets, including BraTS 2018 (brain tumor segmentation), TCGA (cancer survival prediction), and LINCS L1000 (gene expression).

The "Rainbow" Effect and Generalization

A key visual indicator of a well-trained regression model in MDA is the "rainbow" effect: a smooth, continuous transition of colors representing target values. A broken or scrambled rainbow serves as a visual diagnostic for learning failure. Furthermore, MDA reveals a phenomenon similar to "neural collapse" in regression, where features collapse onto a simplified 1D curve as performance peaks. It also provides a clear visual distinction between training and testing sets; well-generalized models maintain a smooth manifold structure in the test set, whereas overfitting results in scattered points.

Quantitative Superiority

Quantitative analysis using Pearson correlation confirms that MDA preserves original high-dimensional geodesic distances significantly better than t-SNE or UMAP, often by a factor of 3x to 4x in complex tasks like survival prediction. MDA also proved sensitive to noise; as the Signal-to-Noise Ratio (SNR) decreased, the MDA "arc" visibly degraded, providing an intuitive measure of model robustness.

Comparison of MDA with t-SNE and UMAP, showing visual manifold preservation (a-c) and quantitative Pearson correlation metrics (d).

Conclusion

MDA represents a significant step toward "Interpretable AI" for regression models. By providing a reliable way to visualize the feature continuum, it allows researchers to assess model robustness, generalizability, and the evolution of features across layers. While computationally more intensive than linear methods, its ability to reveal hidden geometric patterns makes it an essential tool for modern science and medicine.

Summary updated

Abstract

Deep neural networks (DNNs) extract thousands to millions of task-specific features during model training for inference and decision-making. While visualizing these features is critical for comprehending the learning process and improving the performance of the DNNs, existing visualization techniques work only for classification tasks. For regressions, the feature points lie on a high dimensional continuum having an inherently complex shape, making a meaningful visualization of the features intractable. Given that the majority of deep learning applications are regression-oriented, developing a conceptual framework and computational method to reliably visualize the regression features is of great significance. Here, we introduce a manifold discovery and analysis (MDA) method for DNN feature visualization, which involves learning the manifold topology associated with the output and target labels of a DNN. MDA leverages the acquired topological information to preserve the local geometry of the feature space manifold and provides insightful visualizations of the DNN features, highlighting the appropriateness, generalizability, and adversarial robustness of a DNN. The performance and advantages of the MDA approach compared to the existing methods are demonstrated in different deep learning applications.
BibTeX
@inproceedings{islam2023revealing,
  author = {Md Tauhidul Islam and Zixia Zhou and Hongyi Ren and Masoud Badiei Khuzani and Daniel S. Kapp and James Zou and Lü Tian and Joseph C. Liao and Lei Xing},
  title = {{Revealing hidden patterns in deep neural network feature space continuum via manifold learning}},
  booktitle = {Nature Communications},
  year = {2023},
  doi = {10.1038/s41467-023-43958-w},
}
Figure