GeoGNN: Quantifying and Mitigating Semantic Drift in Text-Attributed Graphs

Liangwei Yang, Jing Ma, Jianguo Zhang, Zhiwei Liu, Jielin Qiu, Shirley Kokane, Shiyu Wang, Haolin Chen, Rithesh Murthy, Ming Zhu, Huan Wang, Weiran Yao, Caiming Xiong, Shelby HeineckearXiv 2025 · DOI · arXiv

Introduction: The Geometry of Text

Text-Attributed Graphs (TAGs)—where nodes represent entities with textual descriptions like user bios in social networks or abstracts in citation graphs—are ubiquitous. Typically, these graphs are processed by first encoding text into vectors using Pretrained Language Models (PLMs) like BERT and then applying Graph Neural Networks (GNNs) to aggregate neighborhood information.

However, a fundamental geometric mismatch exists. Standard GNNs perform linear, Euclidean averaging of neighbor features. Recent research suggests that language embeddings do not live in a flat Euclidean space but rather on highly non-linear, curved semantic manifolds.

Conceptual illustration of Semantic Drift: Euclidean averaging (n4) pulls representations off the manifold, whereas geodesic aggregation (n3) preserves semantic integrity.

When a GNN "cuts the corner" of this curve via linear averaging, it pulls the resulting representation into a vacuum where the original semantic meaning is distorted. The authors term this phenomenon Semantic Drift. As GNN layers increase, this drift accumulates, effectively destroying the expressive power of the underlying language model.

Quantifying Semantic Drift

To measure this effect, the authors introduced a local PCA-based metric. For a given node, they estimate the "local tangent plane" of the semantic manifold using its neighbors in the original PLM space. They then calculate how far a GNN’s aggregated vector deviates from this plane.

Average drift across GNN layers for different smoothing methods. Geodesic aggregation (GeoGNN) maintains low drift compared to standard methods.

Empirical results show that in standard GNNs (Mean, Laplacian, or Attention-based), drift scores increase significantly with each layer. By the third or fourth layer, representations are often entirely "off-manifold," leading to a loss of linguistic nuance.

The GeoGNN Framework: Geodesic Aggregation

To mitigate drift, the authors propose GeoGNN, a framework that replaces Euclidean operations with manifold-aware geodesic aggregation.

1. Manifold Normalization

GeoGNN projects embeddings onto a unit hypersphere. This serves as a stable and mathematically tractable proxy for the complex semantic manifold, allowing for efficient Riemannian computations.

2. Log-Exp Mappings

Instead of direct vector addition, GeoGNN utilizes:

  • Log Map: Projects neighbor embeddings into a flat tangent space relative to the target node.
  • Exp Map: After performing aggregation in the tangent space, the result is "wrapped" back onto the curved manifold.

3. Geodesic Attention

GeoGNN introduces a novel attention mechanism grounded in the encoder's original semantic space. It uses geodesic proximity (cosine similarity on the sphere) to weight neighbors, ensuring the graph structure respects the linguistic relationships established by the PLM.

Experimental Results

GeoGNN was evaluated on the CS-TAG benchmark (Photo, Children, Arxiv, and History datasets) across various text encoders (RoBERTa, BERT, MiniLM, Sentence-T5).

  • Node Classification: GeoGNN improved accuracy by 2–3% on average. Gains were highest on datasets with diverse or noisy text, such as "Children" (+3.2%).
  • Link Prediction: The model achieved a 4–6% improvement in Hit@10 scores, demonstrating that preserving local geometry is vital for identifying potential connections.
  • Robustness: GeoGNN exhibited significantly lower variance across different text encoders compared to baselines, indicating it more effectively preserves the intent of the underlying PLM.

Conclusion

GeoGNN demonstrates that the common practice of neighborhood smoothing in GNNs can be a form of geometric distortion for text data. By acting as a "steward" of the language model's features through geodesic aggregation, GeoGNN allows models to leverage graph structures without sacrificing semantic integrity. Future work may explore adaptive curvature modeling to better capture the variable geometry of semantic spaces.

Summary updated

Abstract

Graph neural networks (GNNs) on text--attributed graphs (TAGs) typically encode node texts using pretrained language models (PLMs) and propagate these embeddings through linear neighborhood aggregation. However, the representation spaces of modern PLMs are highly non--linear and geometrically structured, where textual embeddings reside on curved semantic manifolds rather than flat Euclidean spaces. Linear aggregation on such manifolds inevitably distorts geometry and causes semantic drift--a phenomenon where aggregated representations deviate from the intrinsic manifold, losing semantic fidelity and expressive power. To quantitatively investigate this problem, this work introduces a local PCA--based metric that measures the degree of semantic drift and provides the first quantitative framework to analyze how different aggregation mechanisms affect manifold structure. Building upon these insights, we propose Geodesic Aggregation, a manifold--aware mechanism that aggregates neighbor information along geodesics via log--exp mappings on the unit sphere, ensuring that representations remain faithful to the semantic manifold during message passing. We further develop GeoGNN, a practical instantiation that integrates spherical attention with manifold interpolation. Extensive experiments across four benchmark datasets and multiple text encoders show that GeoGNN substantially mitigates semantic drift and consistently outperforms strong baselines, establishing the importance of manifold--aware aggregation in text--attributed graph learning.
BibTeX
@misc{yang2025geognn,
  author = {Liangwei Yang and Jing Ma and Jianguo Zhang and Zhiwei Liu and Jielin Qiu and Shirley Kokane and Shiyu Wang and Haolin Chen and Rithesh Murthy and Ming Zhu and Huan Wang and Weiran Yao and Caiming Xiong and Shelby Heinecke},
  title = {{GeoGNN: Quantifying and Mitigating Semantic Drift in Text-Attributed Graphs}},
  year = {2025},
  eprint = {2511.09042},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.09042},
}
Figure