Abstract
Principal Component Trees (PCTs) are a novel graph-based generalization of Principal Component Analysis (PCA) designed to represent complex, intersecting, and hierarchical subspace structures in high-dimensional data. By combining recursive hypothesis testing with persistent homology, PCTs provide a powerful topological signature to compare the structural complexity of datasets, such as neural network latent spaces and word embeddings.
Introduction
Most dimensionality reduction techniques assume data lies either in a single flat subspace (PCA) or in a union of multiple disjoint subspaces (Subspace Clustering). However, real-world data frequently contains features shared across groups alongside unique cluster-specific features. Standard methods struggle to represent these "intersecting unions of subspaces" efficiently.
Principal Component Trees (PCTs) address this gap by proposing a hierarchical structure where each node represents a principal component and branches represent the different ways those components can be mixed to approximate the data. This allows for a more nuanced representation of data that doesn't fit neatly into disjoint clusters.
Methodology
The core of the PCT framework is the recursive construction of a tree that splits principal components rather than just data points. The algorithm starts at a root node and greedily expands by analyzing residuals—the variance not yet explained by PCA at that level.
To decide whether to split a node into children (indicating a cluster) or maintain it as a single unit, the authors employ a non-parametric Cramér-von Mises hypothesis test. This test evaluates the distribution of pairwise angles between data points; "crowded" angles suggest the presence of subspace clustering. A significant innovation is the intersection test, which prevents premature splitting if two subspaces share a common axis, allowing the PCT to explicitly model intersections.
Furthermore, the authors utilize tools from topology—specifically persistent homology—to define "Height" and "Width" curves. These monotonic signatures describe how the tree's shape evolves as less significant nodes are pruned. The Area Under the Curve (AUC) for these signatures serves as a single-number summary of a dataset's structural complexity or "clusteredness."
Key Insights and Results
The authors demonstrate that if data follows a multivariate normal distribution (no clusters), the PCT naturally reduces to standard PCA, resulting in a tall, thin "degenerate" tree. In contrast, complex datasets like Fashion MNIST exhibit significantly more subspace clustering than standard MNIST. While both have similar dimensions, the MNIST tree is dominated by a single branch, whereas the Fashion MNIST tree splits into two, reflecting its more intricate category structure.
In the realm of neural networks, PCT analysis confirms that classification models create much more clustered latent spaces than generative models like Autoencoders or VAEs. This supports the intuition that classifiers work by forcing data into distinct, separable subspaces. Additionally, the Width Curve ($W_T$) proved to be a highly sensitive measure, clearly distinguishing the isotropic pressure of a VAE (which had the lowest AUC) from the structure of a standard Autoencoder.
In NLP applications, the study found that Russian word embeddings are significantly more clustered than Italian or English ones, suggesting that different languages may have distinct geometric structures for representing common vocabulary.
Conclusion
Principal Component Trees offer a parameter-efficient and topologically grounded approach to modeling high-dimensional data. By capturing intersecting and hierarchical subspace structures, they provide deeper insights into the geometry of latent spaces and linguistic embeddings. Future work may explore non-binary splits and the integration of PCT structures directly into hierarchical neural network architectures to further leverage these geometric insights.
Summary updated