Introduction: The Geometry of Text
Text-Attributed Graphs (TAGs)—where nodes represent entities with textual descriptions like user bios in social networks or abstracts in citation graphs—are ubiquitous. Typically, these graphs are processed by first encoding text into vectors using Pretrained Language Models (PLMs) like BERT and then applying Graph Neural Networks (GNNs) to aggregate neighborhood information.
However, a fundamental geometric mismatch exists. Standard GNNs perform linear, Euclidean averaging of neighbor features. Recent research suggests that language embeddings do not live in a flat Euclidean space but rather on highly non-linear, curved semantic manifolds.
When a GNN "cuts the corner" of this curve via linear averaging, it pulls the resulting representation into a vacuum where the original semantic meaning is distorted. The authors term this phenomenon Semantic Drift. As GNN layers increase, this drift accumulates, effectively destroying the expressive power of the underlying language model.
Quantifying Semantic Drift
To measure this effect, the authors introduced a local PCA-based metric. For a given node, they estimate the "local tangent plane" of the semantic manifold using its neighbors in the original PLM space. They then calculate how far a GNN’s aggregated vector deviates from this plane.
Empirical results show that in standard GNNs (Mean, Laplacian, or Attention-based), drift scores increase significantly with each layer. By the third or fourth layer, representations are often entirely "off-manifold," leading to a loss of linguistic nuance.
The GeoGNN Framework: Geodesic Aggregation
To mitigate drift, the authors propose GeoGNN, a framework that replaces Euclidean operations with manifold-aware geodesic aggregation.
1. Manifold Normalization
GeoGNN projects embeddings onto a unit hypersphere. This serves as a stable and mathematically tractable proxy for the complex semantic manifold, allowing for efficient Riemannian computations.
2. Log-Exp Mappings
Instead of direct vector addition, GeoGNN utilizes:
- Log Map: Projects neighbor embeddings into a flat tangent space relative to the target node.
- Exp Map: After performing aggregation in the tangent space, the result is "wrapped" back onto the curved manifold.
3. Geodesic Attention
GeoGNN introduces a novel attention mechanism grounded in the encoder's original semantic space. It uses geodesic proximity (cosine similarity on the sphere) to weight neighbors, ensuring the graph structure respects the linguistic relationships established by the PLM.
Experimental Results
GeoGNN was evaluated on the CS-TAG benchmark (Photo, Children, Arxiv, and History datasets) across various text encoders (RoBERTa, BERT, MiniLM, Sentence-T5).
- Node Classification: GeoGNN improved accuracy by 2–3% on average. Gains were highest on datasets with diverse or noisy text, such as "Children" (+3.2%).
- Link Prediction: The model achieved a 4–6% improvement in Hit@10 scores, demonstrating that preserving local geometry is vital for identifying potential connections.
- Robustness: GeoGNN exhibited significantly lower variance across different text encoders compared to baselines, indicating it more effectively preserves the intent of the underlying PLM.
Conclusion
GeoGNN demonstrates that the common practice of neighborhood smoothing in GNNs can be a form of geometric distortion for text data. By acting as a "steward" of the language model's features through geodesic aggregation, GeoGNN allows models to leverage graph structures without sacrificing semantic integrity. Future work may explore adaptive curvature modeling to better capture the variable geometry of semantic spaces.
Summary updated