For decades, the dominant paradigm for processing spatial data—whether 2D images, 3D molecular structures, or multi-view video streams—relied on hardcoded geometric inductive biases. Engineers meticulously designed convolutional layers for translation invariance, SE(3)-equivariant networks to handle molecular rotations, and epipolar geometry constraints to align camera views. The underlying assumption was that neural networks needed explicit mathematical rules to navigate the physical world.
However, a powerful counter-trend has emerged. Instead of forcing networks to obey rigid geometric constraints, researchers are discovering that high-capacity, general-purpose sequence models—such as Transformers and State Space Models—can implicitly learn spatial geometry directly from data. By replacing specialized architectural priors with global attention mechanisms and scaling up the data, these models naturally internalize complex 3D structures, physical distances, and view-dependent consistencies. They do not just process geometry; they discover it.
Molecular Grammar and Emergent 3D Conformations
How can a model understand the three-dimensional shape of a molecule if it is only trained on text? In molecular machine learning, text-based representations like SMILES strings flatten 3D structures into 1D sequences. Intuitively, one might expect a model trained on such data to be blind to spatial geometry.
Yet, Yair Schiff et al. demonstrated in Characterizing the Latent Space of Molecular Deep Generative Models with Persistent Homology Metrics using topological data analysis that deep generative models trained purely on 1D chemical strings naturally organize their latent spaces to reflect 3D structural information. This happens because the textual grammar of a molecule—the sequence of its bonds and functional groups—is fundamentally constrained by physical laws. To predict or edit these strings accurately, a model must internalize the underlying spatial relationships. For instance, Tianhao Peng et al. in Pre-trained molecular language models with random functional group masking and Kangjie Zheng et al. in SMI-Editor: Edit-based SMILES Language Model with Fragment-level Supervision showed that sequence-based pre-training objectives (such as fragment or functional group masking) force 1D language models to implicitly capture 3D spatial distances, often outperforming models that explicitly use 3D spatial data.
This shift from explicit to implicit geometry is even more pronounced in 3D molecular design and structural biology. Traditional models rely on rigid SE(3)-equivariant layers to ensure that a protein or molecule's representation remains invariant to rotation and translation. While mathematically elegant, these constraints introduce massive computational overhead. Artem Zholus et al. challenged this paradigm with BindGPT: A Scalable Framework for 3D Molecular Design via Language Modeling and Reinforcement Learning, showing that standard decoder-only transformers can learn complex 3D spatial relationships and physics directly from coordinate data represented as text, bypassing hardcoded symmetries entirely. Similarly, Axel Elaldi et al. utilized a Mamba-based sequence model in Bio2Token: All-atom tokenization of any biomolecular structure with Mamba, relying on extensive data augmentation rather than rigid structural constraints to achieve sub-Angstrom reconstruction fidelity.
By swapping architectural constraints for raw capacity, models like those proposed by Rohit Dilip et al. in Flow Autoencoders are Effective Protein Tokenizers and Shaoning Li in ProteinAE: Protein Diffusion Autoencoders for Structure Encoding demonstrate that standard attention mechanisms and flow-matching objectives can effectively learn 3D structural representations. Data augmentation teaches the model that a rotated molecule is the same molecule, while the attention mechanism dynamically routes information between atoms to compute physical distances, stabilizing training and scaling to massive molecular systems.
Replacing Camera Matrices with Cross-Attention
In computer vision, synthesizing a novel view of a 3D scene from 2D images traditionally requires explicit geometric proxies: camera calibration matrices, depth maps, or epipolar warping layers. If you do not know where the cameras are, you cannot project the pixels into 3D space.
Implicit geometric learning completely flips this problem. Instead of using explicit geometric projection formulas, a pose-free architecture leverages global self-attention across 2D latent features to learn 3D consistency implicitly. Shitao Tang et al. demonstrated this with MVDiffusion++: A Dense High-resolution Multi-view Diffusion Model for Single or Sparse-view 3D Object Reconstruction, showing that pixels across different views can communicate directly via attention to maintain dense, high-resolution consistency without requiring explicit camera metadata.
Why does this work? Consider two unposed images of the same object from different angles. A standard cross-attention mechanism computes the correlation between every pixel in the first image and every pixel in the second image. As the model trains on massive datasets of 3D-consistent scenes, it discovers that certain visual features always move together along predictable trajectories. The attention mechanism effectively learns to trace out epipolar lines and point correspondences purely from feature correlations.
This emergent spatial awareness allows models to bypass traditional 3D representations entirely. Robin Rombach et al. showed in Geometry-Free View Synthesis: Transformers and no 3D Priors that transformers can "imagine" high-fidelity novel views from a single image without explicit depth maps or point clouds, while Vitor Campagnolo Guizilini et al. leveraged large-scale transformer-based diffusion in Zero-Shot Novel View and Depth Synthesis with Multi-View Geometric Diffusion to render novel views and depth maps zero-shot across diverse environments. Remarkably, Haoru Wang et al. found in The Less You Depend, The More You Learn: Synthesizing Novel Views from Sparse, Unposed Images without Any 3D Knowledge that bias-driven models relying on handcrafted representations hit a performance ceiling as data scales; their pose-free model, UP-LVSM, actually outperformed models supplied with perfect ground-truth camera poses by learning geometry directly from unposed images.
This geometry-agnostic approach extends to multi-view image compression and pose estimation. Xinjie Zhang et al. developed LDMIC: Learning-based Distributed Multi-view Image Coding, a distributed image coding framework where independent cameras compress images without communicating. The joint decoder uses cross-attention to exploit multi-view redundancies without knowing camera positions or focal lengths, making it highly robust for unstructured sensor arrays. Similarly, Wanruo Zhang et al. achieved calibration-free 3D human pose estimation in SVTformer: Spatial-View-Temporal Transformer for Multi-View 3D Human Pose Estimation, lifting 2D projections into 3D structures purely through data-driven attention rather than explicit geometric triangulation.
The Power of General-Purpose Architectures
The success of implicit geometric learning points to a broader philosophical shift in AI: general-purpose architectures plus data scale can substitute for specialized architectural priors.
This was vividly illustrated by Alec Radford et al. in Generative Pretraining From Pixels, which proved that a sequence Transformer trained to predict flattened pixel sequences auto-regressively could learn high-quality visual hierarchies and spatial structures without any 2D convolutional biases. As Andrew Shin et al. noted in Perspectives and Prospects on Transformer Architecture for Cross-Modal Tasks with Language and Vision, global self-attention can effectively learn the spatial relationships previously hard-coded into convolutional architectures given sufficient data and capacity.
When applied to 3D data, this principle allows standard architectures to outperform highly specialized geometric pipelines. For example, Ishan Misra et al. introduced An End-to-End Transformer Model for 3D Object Detection, an end-to-end transformer for 3D object detection in point clouds. By replacing complex, hand-tuned 3D geometric operators with standard self-attention, they outperformed highly optimized baselines while keeping the model conceptually simple. Furthermore, Rui Yang et al. demonstrated with Visual Spatial Tuning that general-purpose vision-language models can acquire human-like 3D perception and reasoning abilities purely through large-scale exposure to diverse geometric data, completely eliminating the need for specialized 3D encoders.
Ultimately, implicit geometric learning works because geometry is not an arbitrary set of rules to be programmed; it is a highly structured regularity inherent in the data itself. Whether it is the physical distance between two carbon atoms or the perspective shift of a camera, these geometric truths leave unmistakable statistical footprints. High-capacity attention mechanisms excel at finding these footprints, proving that sometimes the best way to model the physical world is to let the model discover it on its own.
Key Foundational Work
Generative Pretraining From Pixels (2020) — contributed 1 concept elements to this direction.
Characterizing the Latent Space of Molecular Deep Generative Models with Persistent Homology Metrics (2020) — contributed 1 concept elements to this direction.
Image Transformer (2018) — contributed 1 concept elements to this direction.