Higher Order Tensors for Geometric Deep Learning

23 concepts17 papers

When deep neural networks process 3D data—whether mapping the coordinates of a protein, reconstructing a scanned room, or predicting the forces on an atom—they must understand physical space. Early geometric models simplified this problem by relying primarily on scalar invariants (such as distances and angles) or first-order vectors (such as relative displacements).

While scalars and vectors are easy to compute, they possess a fundamental limitation: they struggle to cleanly represent how a space bends, twists, or covaries. To capture complex geometric properties like local surface curvature, directional bonding, or multi-body physical interactions, recent research has shifted toward a more powerful mathematical framework: higher-order geometric tensors.

By allowing hidden layers to pass and transform higher-rank tensors (such as matrices, rank-2 tensors, and higher-order irreducible representations), these architectures gain a native vocabulary for complex spatial structures.

The Limitation of Low-Order Features

To understand why higher-order representations are necessary, consider the task of describing a point on a smooth 3D surface.

A scalar (rank-0 tensor) can tell you the local depth or thickness. A vector (rank-1 tensor) can point in the direction of the surface normal. However, neither can describe curvature—how the surface normal changes as you move across the surface. Curvature is inherently a second-order property; it describes a relationship between directions, traditionally captured by a $3 \times 3$ Hessian matrix or a rank-2 tensor.

When a network is restricted to scalars and vectors, it must attempt to approximate these higher-order properties indirectly, often requiring dense neighborhoods of points and massive amounts of training data. As Andreas Burger et al. point out in their work on Hessian Interatomic Potentials without derivatives, standard equivariant networks that focus only on scalar and vector features lack the necessary mathematical machinery to represent the directional curvature of atomic interactions.

Why Higher-Order Tensors Work: Intuition and Mechanics

Higher-order geometric tensors solve this by explicitly preserving directional context across layers. Instead of collapsing vector interactions into scalar distances, these networks use tensor products to combine lower-order features into higher-order ones.

Rank-0: Scalar (Distance, Mass)  ---> [ 5.0 ]
Rank-1: Vector (Direction, Force) ---> [ x, y, z ]
Rank-2: Tensor (Curvature, Stress) ---> [ xx, xy, xz ]
                                        [ yx, yy, yz ]
                                        [ zx, zy, zz ]

1. Preserving Rotational Equivariance

If you rotate a molecule or a 3D point cloud, its internal geometric properties—like its principal curvature axes or its covariance matrix—must rotate accordingly. This property is known as equivariance.

When we compute a tensor product between two vectors, we generate a higher-order tensor that transforms predictably under rotation groups like $SO(3)$ or $E(3)$. As demonstrated by Sourya Basu et al. in G-RepsNet: A Lightweight Construction of Equivariant Networks for Arbitrary Matrix Groups, utilizing second-order tensors rather than simple vectors significantly enhances a network's ability to handle complex spatial transformations and rotational symmetries natively.

2. Drastic Increases in Data Efficiency

Baking geometry directly into the tensor structure acts as a powerful inductive bias. The network no longer needs to see an object from thousands of different angles to learn how its features rotate.

In E(3)-equivariant graph neural networks for data-efficient and accurate interatomic potentials, Simon Batzner et al. showed that representing atomic environments using higher-order geometric tensors allowed their model (NequIP) to achieve state-of-the-art accuracy using up to three orders of magnitude less training data than scalar-only architectures. Similarly, Lucas Tecot et al. found that incorporating higher-order tensors into hidden layers significantly improves data efficiency when predicting complex quantum electron densities in A recipe for cracking the quantum scaling limit with machine learned electron densities.

3. Scaling Expressive Power

How much more powerful is a higher-order network? Chaitanya K. Joshi et al. provided a formal answer in On the Expressive Power of Geometric Graph Neural Networks. They proved that increasing the tensor order (using higher-order spherical tensors) directly scales a model's ability to resolve complex rotational symmetries that lower-order models find indistinguishable.

This theoretical scaling matches empirical findings by Ben Finkelshtein et al. in A Simple and Universal Rotation Equivariant Point-Cloud Network, which demonstrated that increasing the maximal tensor order predictably enhances a network's capacity to represent complex geometric descriptors, such as the eigenvalues of a point cloud's covariance matrix.


Application Domains

The shift toward higher-order geometric features is driving breakthroughs across two primary fields: 3D computer vision and molecular modeling.

3D Vision and Shape Reconstruction

In computer vision, extracting second-order geometric features allows models to identify fine-grained structural topologies that depth maps alone miss. Early work by Thanuja Dharmasiri et al. in Joint Prediction of Depths, Normals and Surface Curvature from RGB Images using CNNs highlighted that surface curvature maps act as high-fidelity descriptors for object edges and saddle points.

More recently, this insight has been integrated directly into deep architectures:

Molecular Dynamics and Quantum Chemistry

In chemistry and materials science, physical properties are governed by complex many-body interactions and directional bonding. Standard graph neural networks that only look at pairwise distances fail to capture the full quantum-mechanical picture.

The Computational Frontier

If higher-order tensors are so expressive, why hasn't the entire deep learning field adopted them? The bottleneck is computational complexity. Computing tensor products—especially when scaling to higher degrees of irreducible representations—historically incurred steep computational overhead.

To combat this, researchers are focusing on optimizing the underlying tensor algebra. For example, Shengjie Luo et al. introduced a major efficiency breakthrough in Enabling Efficient Equivariant Operations in the Fourier Basis via Gaunt Tensor Products. By optimizing equivariant operations in the Fourier basis, they demonstrated a significant reduction in computational cost, allowing models to scale to higher-order tensors and achieve a 16.8% relative improvement in energy and force metrics on large-scale catalyst datasets.

By resolving these computational bottlenecks, higher-order geometric tensors are transitioning from a theoretical luxury to a practical necessity, providing neural networks with the native mathematical depth required to truly understand the 3D world.

Key Foundational Work

High Order Eigentensors as Symbolic Rules in Competitive Learning (1998) — contributed 2 concept elements to this direction.
Joint Prediction of Depths, Normals and Surface Curvature from RGB Images using CNNs (2017) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_higher_order_geometric_features_d420d853,
  title = {higher-order geometric features},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/higher-order-geometric-features/rdr_d420d853093e4047a210612037949f03},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_higher_order_geometric_features_d420d853_v1,
  title = {higher-order geometric features},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/higher-order-geometric-features/rdr_d420d853093e4047a210612037949f03/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}

See something wrong? Report a mistake