VBA: Vector Bundle Attention for Intrinsically Geometric Representation Learning

Shenglei Fang, Xianfang Sun, You ZhouICML 2026 · OpenReview

TL;DR

This paper introduces the Vector Bundle Attention Transformer (VBA-Transformer), a novel architecture that fundamentally redefines the attention mechanism to intrinsically incorporate geometric information. By treating data points as elements of a learned vector bundle, VBA aligns features using parallel transport before computing similarity, leading to state-of-the-art performance on challenging geometric learning tasks like single-cell RNA sequencing and spatial transcriptomics.

What Problem Does This Paper Address?

Many real-world datasets, from biological tissues to 3D objects, possess inherent geometric structures. Effectively learning from such data requires not only understanding feature similarities but also how these features are aligned in space. Traditional methods like Graph Neural Networks (GNNs) excel at local structure but struggle with long-range dependencies, while standard Transformers are "geometry-blind," treating all data points as fully connected. Existing attempts to incorporate geometry often treat it as an external bias or positional encoding, failing to integrate it deeply into the core similarity computation. This paper addresses the challenge of creating a representation learning framework that natively embeds and leverages intrinsic geometric properties within the attention mechanism itself.

Methods

The core innovation is the Vector Bundle Attention (VBA) mechanism. Instead of treating tokens as simple feature vectors, VBA represents each token as a pair: a base manifold coordinate and a fiber feature vector. This aligns with vector bundle theory, where each point on a base manifold has an associated fiber space.

Key components include:

  • Projection to Vector Bundle: Input features are linearly projected into base coordinates and multiple candidate fiber vectors. A gating mechanism combines these to form the final fiber representation.
  • Learnable Connection and Parallel Transport: A crucial step is aligning fiber features across different tokens. This is achieved through a learnable connection field and an isometric transport operator (Tji). This operator, parameterized by a neural network (TransportNet), ensures that features are rotated into a common local coordinate system before similarity is computed, preserving feature lengths.
  • Curvature Correction: To handle non-flat geometries, VBA optionally incorporates a curvature-informed modulation of fiber features. This uses a learned connection field to compute a curvature tensor, which then adjusts the fiber vectors to account for local manifold deviations.
  • Geometry-Consistent Attention: The attention score is computed using the transported keys and queries in the aligned fiber space. This ensures that similarity is evaluated in a geometry-consistent manner, making the attention mechanism intrinsically geometric.

Datasets

The VBA-Transformer was evaluated on several datasets across different domains:

  • Spatial Transcriptomics (ST):
    • Human dorsolateral prefrontal cortex (DLPFC) dataset (Maynard et al., 2021)
    • 10x Genomics human breast cancer Visium dataset (Cui et al., 2025)
  • Single-Cell RNA Sequencing (scRNA-seq):
    • PBMC (Zheng et al., 2017) and Pancreas datasets (Baron et al., 2016; Muraro et al., 2016; Xin et al., 2016; Segerstolpe et al., 2016; Lawlor et al., 2017)
    • Human Cell Landscape (HCL) dataset (Han et al., 2020) for pretraining.
  • 3D Point Clouds:
    • ModelNet40 (Wu et al., 2015)
    • ScanObjectNN (PB-T50-RS split) (Uy et al., 2019)
  • Molecular Property Prediction:
    • QM9 dataset (Ramakrishnan et al., 2014)

Key Insights and Ideas

  • Geometry as an Intrinsic Operator: The most significant insight is redefining attention not as a similarity measure in an ambient space, but as an operator that intrinsically incorporates geometry. By aligning features before comparison, VBA ensures that similarity is computed within a consistent geometric framework.
  • Vector Bundles for Representation: The paper demonstrates the power of vector bundle theory as a principled mathematical framework for disentangling global position (base manifold) from local feature representation (fiber space) in a learned model.
  • Parallel Transport as Frame Alignment: The learned parallel transport operator acts as an explicit frame-alignment mechanism, making attention more robust to geometric perturbations and variations, especially in complex 3D or biological data.
  • Curvature for Non-Flat Geometries: Incorporating curvature correction allows the model to adapt to and effectively learn from non-Euclidean manifolds, a crucial capability for many real-world datasets.
  • Generalizability Beyond Bioinformatics: The success across diverse domains (biology, 3D vision) validates VBA's core principles as a generalizable geometric representation learning approach.

Key Results

  • State-of-the-Art in scRNA-seq: VBA-SC achieved state-of-the-art accuracy (80.68% OA on PBMC, 97.64% OA on Pancreas) for cell type annotation, outperforming Transformer baselines by over 3-5%.
  • Superior Clustering in ST: VBA-ST demonstrated superior clustering performance on spatial transcriptomics datasets, achieving the best or tied-for-best ARI on several DLPFC samples and outperforming baselines by a significant margin on the Breast Cancer dataset.
  • Competitive 3D Point Cloud Performance: VBA-P achieved competitive accuracy on ModelNet40 (92.9% OA) and showed strong robustness to rotations, outperforming many attention-based baselines in challenging settings.
  • Robustness and Stability: Theoretical analysis and empirical results demonstrated the invariance and perturbation stability of the VBA mechanism.
  • Interpretability: The learned bundle weights showed clear specialization for different object classes and consistent patterns within classes, offering insights into the model's decision-making.

Limitations and Future Work

  • Computational Complexity: The pairwise transport operator leads to higher computational complexity (O(n²d_f)) compared to standard Transformers (O(n²d)). The authors suggest future work on sparse attention or low-rank parameterizations to mitigate this for very large datasets.
  • Base Manifold Dimension Selection: The choice of base manifold dimension (d_b) is a key hyperparameter. Future research could explore methods for automatically learning or adapting this dimension from data.
  • Path-Dependent Transport: The current endpoint-dependent transport is a surrogate for true path-dependent parallel transport. Exploring more faithful path integration methods is a promising direction.
  • Generalization to Other Geometric Tasks: While successful, the authors suggest further comparisons with specialized geometric architectures to clarify VBA's scope and advantages.

Summary updated

Abstract

Learning from geometrically structured data is central to applications in biology, physics, and computer vision. In many tasks, meaningful comparisons depend on how features are aligned in space. Graph Neural Networks capture local structure but are constrained by message passing. Transformers model long-range dependencies but largely ignore geometry. We introduce the Vector Bundle Attention Transformer (VBA-Transformer), a framework that redefines attention as an intrinsic geometric operator. Each token couples a base manifold coordinate with a fiber feature vector, following vector bundle theory. A principled parallel transport mechanism aligns fiber features across local coordinate systems before similarity is computed. This embeds geometry directly into the attention operator. Unlike prior methods that inject geometry as an external bias or positional encoding, VBA integrates geometry natively inside attention. On challenging single-cell RNA sequencing benchmarks, VBA achieves state-of-the-art accuracy, outperforming Transformer baselines by over 3--5\%. On spatial transcriptomics, it demonstrates superior clustering performance. On 3D point clouds, it achieves competitive accuracy, validating broad generalization across domains. Beyond empirical gains, we provide theoretical analysis of invariance and perturbation stability. We also demonstrate robust transport behavior empirically. Together, these results establish intrinsic geometric alignment as a powerful principle for scalable representation learning.
BibTeX
@inproceedings{fang2026vba,
  author = {Shenglei Fang and Xianfang Sun and You Zhou},
  title = {{VBA: Vector Bundle Attention for Intrinsically Geometric Representation Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=nQ4iLQtsCF},
}
Figure