Simplifying Simplex Convolutional Networks

Gabriel Jonas Duarte, Thiago Filipe de Lima Arruda, Diego Mesquita, Amauri H. Souza2024 · DOI

Abstract

Topological deep learning (TDL) has recently emerged as a family of neural networks for data on topological domains (e.g., graphs and cellular complexes). In the context of graph data, TDL provides a recipe for leveraging high-order information, e.g., from cliques, to boost the expressiveness of graph neural networks. This additional power comes at a computational cost, stemming from message passing between higher-order structures. In this paper, we alleviate the computing toll of TDL by proposing SimpleXCN, a linear convolutional model for simplicial complexes. SimpleXCN derives naturally from removing non-linearities of base TDL architectures. Despite its simplicity, SimpleXCN remains competitive with established topological neural networks in node classification tasks. Our experiments show a reduction of approximately 25% in time and 10 times in the number of parameters, and consequently a significant reduction in memory usage of up to 50% compared to other SNNs.

Introduction

Standard Graph Neural Networks (GNNs) are primarily limited by their pairwise message-passing schemes. While effective for many tasks, they often fail to capture higher-order interactions inherent in complex systems, such as the triadic relationships in social networks or multi-protein complexes in biological data. Topological Deep Learning (TDL), specifically Simplicial Neural Networks (SNNs), addresses this by modeling data on simplicial complexes—mathematical structures that generalize graphs to include triangles, tetrahedrons, and higher-dimensional "simplices."

However, the increased expressivity of SNNs comes with a significant "computational toll." The process of passing messages between these higher-order structures requires substantial memory and processing power, hindering scalability. SimpleXCN addresses this by asking if the complexity of these networks can be stripped away without sacrificing predictive performance.

Methodology

SimpleXCN is inspired by the "Simple Graph Convolution" (SGC) paradigm, extending it to the topological domain. The core idea is to linearize the Simplex Convolutional Network (SCN) architecture.

Visual representation of k-simplices, from nodes (0-simplex) to tetrahedrons (3-simplex).

The methodology involves four key technical steps:

  1. Linearization: By removing non-linear activation functions (like ReLU) between layers, multiple steps of topological diffusion can be collapsed into a single linear operation.
  2. Hodge Laplacians: The model utilizes Hodge Laplacians ($\Delta_r$) to define neighborhoods across different dimensions (vertices, edges, faces), enabling structured information flow.
  3. Feature Lifting: To handle datasets where attributes are only available for nodes, a recursive "lifting" process propagates these features to higher-order simplices.
  4. Pre-computation: Because the model is linear, the diffusion process—the most computationally expensive part—can be pre-computed once. Training then only requires learning a single weight matrix $\Theta_r$ per dimension.

Experimental Setup

The model was evaluated across seven diverse datasets, categorized by their connectivity patterns:

  • Homophilic Citation Networks: Cora, Citeseer, PubMed, and DBLP.
  • Heterophilic Networks: Amazon-ratings, Minesweeper (a grid-based graph), and Questions (a user interaction network).

Key Results and Insights

SimpleXCN demonstrates that for many node classification tasks, the non-linearities in topological layers are often redundant. The "less is more" philosophy pays off significantly in terms of efficiency.

Comparison of parameter count and memory usage across different models and datasets, highlighting the efficiency of SimpleXCN.

Key findings include:

  • Parameter Efficiency: SimpleXCN achieves a massive reduction in model size. For instance, on the DBLP dataset, parameters were reduced from ~173K to just 22.9K.
  • Memory Savings: Memory usage on the "Questions" dataset dropped from nearly 5GB to 0.61GB, a reduction far exceeding initial expectations.
  • Computational Speed: The authors observed an average 25% reduction in execution time per iteration.
  • Competitive Accuracy: Despite its simplicity, SimpleXCN remained statistically competitive with state-of-the-art models like SCN and SCCN, even outperforming them in specific cases like the Citeseer dataset.

Discussion and Future Work

While SimpleXCN is highly efficient for node classification, future research is needed to determine if this linear simplification remains effective for more complex tasks like edge prediction or the classification of higher-order simplices themselves. Additionally, the current study primarily utilized "clique lifting" up to 2-simplices (triangles); exploring higher-order structures and extending this simplification to cellular complexes or hypergraphs represents a promising frontier for topological deep learning.

Summary updated

Abstract

Topological deep learning (TDL) has recently emerged as a family of neural networks for data on topological domains (e.g., graphs and cellular complexes). In the context of graph data, TDL provides a recipe for leveraging high-order information, e.g., from cliques, to boost the expressiveness of graph neural networks. This additional power comes at a computational cost, stemming from message passing between higher-order structures. In this paper, we alleviate the computing toll of TDL by proposing SimpleXCN, a linear convolutional model for simplicial complexes. SimpleXCN derives naturally from removing non-linearities of base TDL architectures. Despite its simplicity, SimpleXCN remains competitive with established topological neural networks in node classification tasks. Our experiments show a reduction of approximately 25% in time and 10 times in the number of parameters, and consequently a significant reduction in memory usage of up to 50% compared to other SNNs.
BibTeX
@misc{duarte2024simplifying,
  author = {Gabriel Jonas Duarte and Thiago Filipe de Lima Arruda and Diego Mesquita and Amauri H. Souza},
  title = {{Simplifying Simplex Convolutional Networks}},
  year = {2024},
  doi = {10.5753/kdmile.2024.244748},
}
Figure