Abstract
Spectral methods of moments provide a powerful tool for learning the parameters of latent variable models. Despite their theoretical appeal, the applicability of these methods to real data is still limited due to a lack of robustness to model misspecification. In this paper we present a hierarchical approach to methods of moments to circumvent such limitations. Our method is based on replacing the tensor decomposition step used in previous algorithms with approximate joint diagonalization. Experiments on topic modeling show that our method outperforms previous tensor decomposition methods in terms of speed and model quality.
Introduction
Discovering latent structures in high-dimensional data—such as identifying underlying topics in a corpus of documents—is a fundamental task in machine learning. While the Expectation-Maximization (EM) algorithm has long been the standard approach, its iterative nature often leads to slow convergence and a tendency to settle in local optima. Spectral Methods of Moments (MoM) offer a faster, globally convergent alternative by utilizing linear algebraic techniques like tensor decomposition.
However, traditional MoM algorithms suffer from extreme sensitivity to model misspecification. If the assumed number of latent states (e.g., topics) is lower than the true count, the mathematical foundations of these methods often collapse, leading to nonsensical parameter estimates. This paper introduces SIDIWO (Simultaneous Diagonalization based on Whitening and Optimization), a hierarchical MoM framework designed to bridge the gap between the robustness of EM and the efficiency of spectral learning.
The SIDIWO Algorithm
SIDIWO reinterprets the tensor decomposition problem as a hierarchical optimization task. Instead of attempting to find all latent factors simultaneously, it employs a divisive binary tree approach.
Simultaneous Diagonalization
The core of SIDIWO involves two primary steps:
- Whitening: The algorithm uses second-order moments (covariance) to project the data into a lower-dimensional subspace.
- Optimization: Rather than using standard tensor power methods, SIDIWO frames the search for latent factors as a non-convex optimization problem. It seeks to minimize the off-diagonal elements of tensor slices, effectively performing approximate joint diagonalization.
The $l=2$ Trick and Hierarchical Splitting
A key innovation is the "$l=2$ trick." The authors demonstrate that when searching for only two latent states, the complex optimization problem simplifies into a one-dimensional grid search. By recursively applying this binary search, SIDIWO builds a hierarchy of topics. This divisive strategy allows the algorithm to "zoom in" on specific data clusters, filtering out noise that might otherwise overwhelm a "flat" decomposition attempt.
Experimental Evaluation
The authors evaluated SIDIWO against standard MoM techniques, including the Tensor Power Method (TPM) and Alternating Least Squares (ALS), using synthetic and real-world datasets.
Quantitative Performance
On a synthetic topic model with an intrinsic hierarchical structure, SIDIWO demonstrated superior accuracy and computational efficiency. It achieved an Adjusted Rand Index (ARI) of 0.98, outperforming TPM (0.93) and ALS (0.70). Notably, SIDIWO was significantly faster, completing the task in 0.4 seconds compared to 1.2 seconds for TPM and 16 minutes for Random Projections.
Qualitative Insights: NIPS and Wikipedia
The hierarchical approach proved highly effective on real-world data. Using a dataset of NIPS conference papers (1987-2015), SIDIWO successfully mapped the evolution of research fields. The first split clearly separated "Neuroscience and Neural Networks" from "Image Processing and Probabilistic Models."
Similarly, on Wikipedia mathematics articles, the algorithm correctly distinguished between "Pure Mathematics" (e.g., Ring Theory) and "Applied Mathematics" (e.g., Optimization).
Robustness and "Pseudo-centers"
Perhaps the most significant finding is SIDIWO's handling of misspecification. When the number of requested topics is smaller than the actual number, SIDIWO produces "pseudo-centers." These are synthetic representations that act as weighted averages for groups of original topics. Geometric proofs provided by the authors show that the optimization objective naturally aligns these pseudo-centers with clusters of real topics while maintaining orthogonality to others.
Conclusion and Future Work
SIDIWO provides a robust and efficient framework for hierarchical topic modeling, overcoming the traditional fragility of spectral methods. Future research will focus on extending this approach to "overcomplete" models, where the number of latent states exceeds the observable dimensions, and exploring applications in complex domains like healthcare analytics for patient sub-typing.
Summary updated