Layer-to-Layer Knowledge Mixing in Graph Neural Network for Chemical Property Prediction

Teng Jiek See, Daokun Zhang, Mario Boley, David K. ChalmersarXiv 2025 · DOI · arXiv

Introduction

Predicting molecular properties—ranging from drug solubility to quantum energy states—is a fundamental challenge in computational chemistry. While Graph Neural Networks (GNNs) have emerged as the standard for these tasks, researchers often face a trade-off: higher accuracy typically requires increasing model complexity, which leads to greater memory usage and computational overhead. This paper introduces Layer-to-Layer Knowledge Mixing (LKM), a self-knowledge distillation technique that enhances GNN performance without adding parameters or increasing inference time.

The Challenge of Multi-Scale Representations

In a standard GNN, successive layers aggregate information from increasingly distant neighbors (k-hops). Consequently, early layers capture local atomic environments, while deeper layers develop a more global understanding of the molecular structure. Often, these layers operate independently, missing opportunities to leverage the complementary information present across different spatial scales. Traditional knowledge distillation (KD) addresses this by training a small "student" model to mimic a massive "teacher" model, but this process is computationally expensive. LKM bypasses this by allowing a model to effectively teach itself.

Methodology: Layer-to-Layer Knowledge Mixing

The LKM framework treats each layer within the GNN as an "expert" specializing in a specific spatial scale. The core innovation lies in the introduction of an auxiliary loss term that encourages layers to align their hidden representations.

Architecture of the Layer-to-Layer Knowledge Mixing (LKM) framework, showing the integration of task loss and knowledge distillation loss.

As illustrated in the architecture above, the process involves:

  1. Consensus Embedding: Calculating a mean "consensus" embedding across all GNN layers for each atom.
  2. Discrepancy Loss ($L_{KD}$): Minimizing the mean absolute distance (MAD) between individual layer embeddings and the consensus average.
  3. Total Loss Optimization: The final objective function combines the standard task loss ($L_{task}$) with the distillation loss ($L_{KD}$), weighted by a hyperparameter $\gamma$: $L_{total} = L_{task} + \gamma L_{KD}$.

This "mixing" forces the model to integrate local and global topological insights into a more holistic representation. Since the auxiliary loss is only used during training, the model's inference speed and parameter count remain unchanged.

Experimental Setup

The researchers evaluated LKM across three benchmark datasets representing different scales of chemical complexity: QM9 for small organic molecules, MD17 for molecular dynamics trajectories, and Chignolin for larger biomolecular systems.

Summary of datasets used for benchmarking: QM9, MD17, and Chignolin.

Key Results and Discussion

LKM was integrated into three state-of-the-art GNN architectures: DimeNet++, MXMNet, and PAMNet. The results demonstrated significant and consistent performance gains:

  • QM9: Average Mean Absolute Error (MAE) was reduced by up to 9.8%.
  • MD17: Energy prediction errors saw a reduction of up to 45.3% for specific molecules.
  • Chignolin: Energy prediction for this 166-atom peptide system improved by 22.9%.

The study also highlighted that the benefit of LKM scales with model depth; deeper networks provide more "experts" for the distillation process, leading to better generalization. However, the effectiveness of LKM is sensitive to the mixing strength parameter $\gamma$. If $\gamma$ is too high, the model may suffer from over-smoothing, where layers become too similar and lose their scale-specific unique information.

Conclusion

Layer-to-Layer Knowledge Mixing provides a "zero-cost" method for boosting the accuracy of GNNs in chemical property prediction. By leveraging internal self-distillation, LKM allows models to achieve state-of-the-art performance while maintaining computational efficiency during deployment. Future work may explore adaptive mixing mechanisms to dynamically weight layer contributions based on molecular context.

Summary updated

BibTeX
@misc{see2025layer,
  author = {Teng Jiek See and Daokun Zhang and Mario Boley and David K. Chalmers},
  title = {{Layer-to-Layer Knowledge Mixing in Graph Neural Network for Chemical Property Prediction}},
  year = {2025},
  eprint = {2510.20236},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.20236},
}
Figure