How Neural Architectures Affect Deep Learning for Communication Networks?

Yifei Shen, Jun Zhang, Khaled B. Letaief

Abstract

Deep learning is a powerful tool for optimizing communication networks, yet standard architectures like Multi-Layer Perceptrons (MLPs) often struggle with scalability and generalization. This paper provides a rigorous theoretical explanation for why Graph Neural Networks (GNNs) outperform MLPs in these tasks. By exploiting permutation invariance, GNNs converge $O(n \log n)$ times faster and achieve $O(n)$ lower generalization error as the number of network nodes ($n$) increases.

Introduction

The application of deep learning to communication network optimization—such as power control and beamforming—has faced significant hurdles. Early models adapted from computer vision, specifically MLPs and CNNs, frequently exhibit poor scalability and weak generalization. A model trained on a small network often fails when deployed on a larger one, and performance degrades if test conditions (e.g., SNR) deviate from training data.

While empirical evidence suggested GNNs were more effective, a mathematical proof explaining why was missing. This research bridges that gap by theoretically validating how architectural choices impact training and testing performance in communication systems.

Theoretical Framework: The Neural Tangent Kernel

The researchers employ the Neural Tangent Kernel (NTK) framework to analyze network evolution during training. NTK allows the complex, non-convex training process of a neural network to be treated as a simpler convex optimization problem in a functional space.

The core of the analysis centers on Permutation Invariance. In most communication problems, such as the $K$-user interference channel, the identity or ordering of users does not alter the underlying physics. Swapping the data of two users should result in a swapped optimal solution. GNNs are architecturally designed to respect this symmetry, whereas MLPs must laboriously learn it from data.

The Optimization Landscape

The "shape" of the optimization problem is critical. For MLPs, as the number of users ($n$) grows, the optimization landscape becomes increasingly "stretched" (characterized by a high condition number), making it difficult for gradient descent to locate the global minimum. In contrast, the GNN landscape remains relatively stable regardless of $n$.

Optimization landscapes of GNN and MLP as network size increases.

By hard-coding permutation invariance, GNNs drastically reduce the search space for the target function. This architectural prior makes GNNs highly sample-efficient, requiring fewer examples to learn the same underlying physical principles. Interestingly, the study suggests that adding more training data can actually hinder MLP training, as the smallest eigenvalue of the NTK matrix shrinks with dataset size, further slowing convergence.

Empirical Validation and Results

The theoretical findings were validated using simulated $K$-user interference channels for power control. The authors compared MLPs and GNNs across networks with 5 and 20 users, with training data volumes ranging from 1,000 to 20,000 samples.

Convergence of GNN vs MLP for 5 and 20 users.

Key results include:

  • Convergence Speed: GNNs were proven to converge $O(n \log n)$ times faster than MLPs. In 20-user simulations, MLPs struggled to minimize training loss, while GNNs converged almost immediately.
  • Generalization Gap: The theoretical generalization error for MLPs is $O(n)$ times higher than for GNNs.
  • Scalability: In the 20-user power control problem, GNNs achieved near-optimal sum-rates, while MLPs failed to generalize patterns learned from different datasets.

Conclusion and Future Work

While GNNs show clear superiority for permutation-invariant tasks, the authors note a "No Free Lunch" caveat: if a task requires distinguishing between specific unique nodes, GNNs may underperform.

Future research could extend this NTK-based analysis beyond two-layer models to deeper architectures or "unrolled" neural networks that incorporate more domain-specific structures. Additionally, exploring symmetries beyond permutation invariance, such as equivariance, remains a promising direction for communication network optimization.

Summary updated

BibTeX
@inproceedings{shen2022how,
  author = {Yifei Shen and Jun Zhang and Khaled B. Letaief},
  title = {{How Neural Architectures Affect Deep Learning for Communication Networks?}},
  booktitle = {ICC},
  year = {2022},
  doi = {10.1109/icc45855.2022.9839205},
}
Figure