Abstract
Equivariant neural networks (ENNs) are typically designed for "perfect" symmetries, where the group action on the input is known exactly. However, real-world data often exhibits "latent" symmetries—transformations that are simple in object space but complex in pixel space due to perspective or occlusions. This paper demonstrates that ENNs are remarkably robust to such imprecisions. By distinguishing between extrinsic and incorrect equivariance, the authors show that imposing approximate symmetry constraints significantly improves sample efficiency and performance in robotics and control tasks, even when the true symmetries are obscured.
The Challenge of Latent Symmetry
In deep learning, equivariance serves as a powerful inductive bias. By baking known task symmetries (e.g., rotation or reflection) into the network architecture, models can achieve better generalization and higher sample efficiency. However, a significant hurdle exists: standard equivariant models require precise knowledge of how the symmetry group acts on input pixels.
In practical scenarios, such as a robot viewing a workspace from an oblique angle, an object's rotation does not translate to a simple image rotation. Instead, it involves complex perspective shifts and foreshortening. This is termed latent symmetry. Conventional wisdom suggests that without exact symmetry knowledge, one should revert to unconstrained models like standard CNNs. This research challenges that assumption, suggesting that approximate image-level transforms can effectively model complex object-level changes.
Categorizing Symmetry Relationships
The authors propose a taxonomy to describe the relationship between a model's assumed symmetry and the ground truth domain:
- Correct Equivariance: The model's symmetry perfectly matches the domain's underlying physics and labels.
- Incorrect Equivariance: The model's symmetry constraints conflict with ground truth labels (e.g., assuming $f(rotate(x)) = f(x)$ when the task requires distinct labels for different orientations).
- Extrinsic Equivariance: The model's symmetry transformation moves data to an "out-of-distribution" (OOD) region. While it doesn't perfectly match the true symmetry, it does not contradict it.
The core hypothesis is that Extrinsic Equivariance provides a helpful inductive bias. Even if an image rotation is an imprecise proxy for an object's rotation, enforcing it regularizes the network, allowing it to learn the underlying latent symmetry more efficiently than an unconstrained baseline.
Experimental Evaluation
The researchers evaluated this hypothesis using Equivariant versions of Soft Actor-Critic (SAC) and DrQv2 across three domains:
- Supervised Rotation Estimation: Identifying relative rotations of objects under corruptions like squeezing and lighting changes.
- BulletArm Robotics Benchmark: Five manipulation tasks (e.g., Block Pulling, Drawer Opening) viewed from fixed, non-top-down angles.
- DeepMind Control Suite (DMC): Seven continuous control tasks where backgrounds and floor grids technically break perfect image-level symmetry.
Key Findings
- Inductive Bias vs. Perfect Modeling: An imprecise equivariant model often outperforms a completely unconstrained one. The extrinsic constraint guides the network toward the correct solution during training.
- The Risk of Incorrect Symmetry: While extrinsic symmetry is beneficial, incorrect symmetry is detrimental. Forcing identical outputs for inputs that require different labels creates a theoretical upper bound on accuracy, which the authors formally prove and empirically validate.
- Sample Efficiency in Robotics: In BulletArm tasks, Equivariant SAC significantly outperformed state-of-the-art baselines like FERM and RAD, achieving higher rewards with fewer interactions despite the lack of "perfect" views.
- Architecture over Augmentation: While data augmentation helps standard CNNs, embedding symmetry directly into the architecture (ENNs) proved consistently more effective for improving sample efficiency.
Conclusion and Future Work
This work reveals that the benefits of equivariance extend far beyond domains with perfect, known symmetries. The "surprising effectiveness" of these models in latent symmetry domains suggests they should be a first-choice architecture even when symmetry is only approximate. Future research may extend these findings to 3D SO(3) symmetries in point clouds and explore methods for the automatic discovery of beneficial extrinsic symmetries in unmodeled domains.
Summary updated