Lie Neurons: Adjoint-Equivariant Neural Networks for Semisimple Lie Algebras

Tzu-Yuan Lin, Minghan Zhu, Maani Ghaffari

Introduction

The modeling of physical systems—ranging from the orientation of satellites to the structural symmetries of molecules—is fundamentally rooted in the mathematics of Lie groups. While deep learning has achieved remarkable success in these domains, standard neural network architectures remain largely "symmetry-blind." Consequently, these models often require extensive data augmentation to achieve invariance or equivariance to transformations like rotations.

To address this, the authors propose Lie Neurons (LN), a novel class of neural networks designed to operate within any finite-dimensional semisimple Lie algebra. By embedding the adjoint representation directly into the network's architecture, Lie Neurons achieve built-in equivariance, significantly enhancing data efficiency and robustness for geometric tasks in robotics, physics, and computer vision.

The Lie Neuron Architecture

The core innovation of Lie Neurons lies in treating data as elements of a Lie algebra and ensuring that every layer commutes with the adjoint action of the corresponding group. This generalizes previous frameworks like Vector Neurons to a broader class of symmetries, including those described by Symplectic ($sp(n)$) or Special Linear ($sl(n)$) algebras.

Integration of Lie Neurons into a Neural ODE framework for dynamics modeling.

Several key components define the LN architecture:

  • Adjoint-Equivariant Linear Layers: These layers operate on the feature dimension while preserving the underlying algebraic structure, ensuring that the geometric properties of the input are maintained throughout the network.
  • LN-ReLU (Killing Form Activation): Traditional activation functions like ReLU are ill-suited for vector-valued data. Lie Neurons utilize the Killing form—a canonical bilinear form on Lie algebras—to determine a "direction" for rectification, providing a geometrically grounded nonlinearity.
  • LN-Bracket (Algebraic Nonlinearity): This novel layer employs the Lie bracket ($[u, v] = uv - vu$) as a structured nonlinearity. Because the bracket is inherently equivariant, it allows the model to capture complex interactions between algebraic elements without breaking symmetry.
  • Geometric Channel Mixing: For specific algebras like $so(n)$, the framework introduces methods to fuse information across geometric dimensions, increasing the model's expressive capacity compared to previous equivariant methods.

Experimental Evaluation

The researchers benchmarked Lie Neurons across a variety of challenging tasks:

  1. Mathematical Regression: Fitting invariant functions on $sp(4)$ and $sl(3)$ algebras.
  2. BCH Formula Learning: Approximating the Baker-Campbell-Hausdorff formula for combining rotations in $so(3)$.
  3. Rigid Body Dynamics: Predicting the tumbling motion of the International Space Station (ISS) using a Neural ODE framework.
  4. Computer Vision: Point cloud registration on ModelNet40 and a Platonic solid classification task involving homography transformations.

Key Results and Robustness

Lie Neurons demonstrated superior performance and robustness across all benchmarks. In $sp(4)$ regression, the model achieved an MSE of $2.70 \times 10^{-4}$, vastly outperforming a standard MLP ($0.107$) despite having fewer parameters.

Performance of Lie Neurons on ISS dynamics: (a) 3D trajectories, (b) learned vector field, and (c) time-series coordinates.

A critical advantage of the LN framework is its zero-shot robustness. In the Platonic solid classification task, while a standard MLP's accuracy plummeted from 95% to 36% when presented with rotated camera views, Lie Neurons maintained a near-perfect 99.6% accuracy. Furthermore, in dynamics modeling, the predicted trajectories for the ISS were qualitatively indistinguishable from ground truth, even when evaluated in entirely new reference frames.

Conclusion and Future Directions

Lie Neurons provide a powerful, general-purpose framework for incorporating geometric priors into neural networks. By leveraging the structural properties of semisimple Lie algebras, these models achieve a level of robustness and data efficiency that is unattainable for symmetry-blind architectures.

Future work may focus on extending the framework to non-semisimple algebras (such as those including translations) and optimizing the "lifting" process—the method by which raw Euclidean data is mapped into the Lie algebra. Additionally, while LN layers are highly efficient in terms of data, reducing the computational overhead of operations like the Lie bracket remains an area for further optimization.

Summary updated

BibTeX
@inproceedings{lin2024lie,
  author = {Tzu-Yuan Lin and Minghan Zhu and Maani Ghaffari},
  title = {{Lie Neurons: Adjoint-Equivariant Neural Networks for Semisimple Lie Algebras}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=1bJLl4fY6i},
}
Figure