Stationary Diffusion State Neural Estimation for Multiview Clustering

Chenghua Liu, Zhuolin Liao, Yixuan Ma, Kun Zhan

Abstract

Although many graph-based clustering methods attempt to model the stationary diffusion state in their objectives, their performance limits to using a predefined graph. We argue that the estimation of the stationary diffusion state can be achieved by gradient descent over neural networks. We specifically design the Stationary Diffusion State Neural Estimation (SDSNE) to exploit multiview structural graph information for co-supervised learning. We explore how to design a graph neural network specially for unsupervised multiview learning and integrate multiple graphs into a unified consensus graph by a shared self-attentional module. The view-shared self-attentional module utilizes the graph structure to learn a view-consistent global graph. Meanwhile, instead of using auto-encoder in most unsupervised learning graph neural networks, SDSNE uses a co-supervised strategy with structure information to supervise the model learning. The co-supervised strategy as the loss function guides SDSNE in achieving the stationary state. With the help of the loss and the self-attentional module, we learn to obtain a graph in which nodes in each connected component fully connect by the same weight. Experiments on several multiview datasets demonstrate effectiveness of SDSNE in terms of six clustering evaluation metrics.

Introduction

Clustering data from multiple sources—such as combining text, images, and metadata—is a powerful but complex task. Traditionally, researchers have relied on graph-based methods to find a consensus between these different "views." However, these methods often depend on rigid, predefined graphs and computationally expensive operations like eigen-decomposition, which are difficult to integrate into modern deep learning pipelines.

Presented at AAAI-22, Stationary Diffusion State Neural Estimation (SDSNE) rethinks this problem. By treating the search for a "stationary diffusion state" as a task for a neural network, SDSNE bypasses traditional spectral solvers and learns a high-quality consensus graph across multiple data views through gradient descent.

The SDSNE Framework

The core innovation of SDSNE is modeling the estimation of the stationary diffusion state as a neural layer. Instead of treating diffusion as a fixed mathematical iteration, the framework uses a parameter-shared self-attentional module to handle multiple views. By sharing a weight matrix across all views, the model is forced to find a view-consistent global structure.

The "Water Cup" Analogy

To understand the stationary state, the authors use a visualization of water flowing between connected cups. In this analogy, connected cups (representing nodes in the same cluster) eventually reach the same water level. SDSNE essentially learns the "pipes" (graph weights) that lead to this balanced, stationary state where nodes in each connected component are fully connected with equal weights.

The "Water Cup" analogy: Diffusion process leading to a stationary state where connected components reach equilibrium.

Unlike many graph neural networks that rely on reconstruction loss (auto-encoders), SDSNE employs a co-supervised strategy. This structural loss function guides the model toward the stationary state by minimizing the "energy" of the graph, pushing the network to learn the cluster labels directly.

Experimental Evaluation

The researchers evaluated SDSNE on six diverse benchmark datasets, including BBC Sport, MSRC-v1, 100 Leaves, Three Sources, Scene-15, and the large-scale Reuters dataset.

Key Results

SDSNE consistently outperformed state-of-the-art (SOTA) methods like GMC, CGD, and O2MAC across all evaluation metrics. Notable highlights include:

  • Significant Accuracy Gains: On the "Three Sources" dataset, SDSNE achieved an accuracy of 93.5%, representing a 15.4% improvement over the next best method.
  • Scalability: The model demonstrated robustness on both small visual datasets and large-scale multilingual textual datasets.
  • Visual Clarity: t-SNE visualizations of the learned representation $H$ revealed much tighter, more distinct clusters compared to raw input features, confirming that the model successfully filters noise to find the underlying consensus structure.
t-SNE visualization of the learned representation, demonstrating clear separation between clusters.

Conclusion

SDSNE provides an end-to-end framework that achieves the benefits of spectral clustering within a deep learning pipeline. While the $O(n^3)$ complexity for matrix multiplication remains a challenge for extremely large-scale applications, the method's ability to learn a view-consistent global graph without iterative eigen-decomposition marks a significant step forward in unsupervised multiview learning. Future work may extend this stationary state estimation to semi-supervised learning and dimensionality reduction tasks.

Summary updated

Abstract

Although many graph-based clustering methods attempt to model the stationary diffusion state in their objectives, their performance limits to using a predefined graph. We argue that the estimation of the stationary diffusion state can be achieved by gradient descent over neural networks. We specifically design the Stationary Diffusion State Neural Estimation (SDSNE) to exploit multiview structural graph information for co-supervised learning. We explore how to design a graph neural network specially for unsupervised multiview learning and integrate multiple graphs into a unified consensus graph by a shared self-attentional module. The view-shared self-attentional module utilizes the graph structure to learn a view-consistent global graph. Meanwhile, instead of using auto-encoder in most unsupervised learning graph neural networks, SDSNE uses a co-supervised strategy with structure information to supervise the model learning. The co-supervised strategy as the loss function guides SDSNE in achieving the stationary state. With the help of the loss and the self-attentional module, we learn to obtain a graph in which nodes in each connected component fully connect by the same weight. Experiments on several multiview datasets demonstrate effectiveness of SDSNE in terms of six clustering evaluation metrics.
BibTeX
@inproceedings{liu2022stationary,
  author = {Chenghua Liu and Zhuolin Liao and Yixuan Ma and Kun Zhan},
  title = {{Stationary Diffusion State Neural Estimation for Multiview Clustering}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=kR3hpvO5XsK},
}
Figure