TL;DR
This paper introduces General Neural Gauge Fields, a framework that allows neural scene representations to learn their own optimal coordinate systems (gauge transformations) instead of relying on fixed, pre-defined ones. By deriving new regularization techniques and invariant functions from the principle of information conservation, the authors prevent the common problem of "gauge collapse" and achieve significantly better rendering quality and efficiency.
What Problem Does This Paper Address?
Modern 3D scene representations, like Neural Radiance Fields (NeRF), often don't work directly with raw 3D coordinates. To speed up training or save memory, researchers use "gauge transformations"—mappings that project 3D points into other systems like 2D tri-planes or discrete hash tables.
However, these mappings are almost always fixed heuristics (e.g., simple orthogonal projections). While they work, they are sub-optimal because they don't adapt to the specific geometry of a scene. The obvious solution is to learn the mapping alongside the scene itself. The catch? Naively learning these transformations leads to gauge collapse: the model finds a "lazy" solution where it maps many different 3D points to the same target coordinate, effectively losing the information needed to reconstruct the scene.
Methods
The authors propose a unified framework for learning both continuous (e.g., mapping to a 2D plane) and discrete (e.g., mapping to a codebook of vectors) gauge transformations. To solve the collapse problem, they introduce two key innovations:
- InfoReg (Information Regularization): Derived from the principle of information conservation, this loss function maximizes the mutual information between the original 3D points and the new target coordinates.
- For continuous cases, they use Earth Mover’s Distance (EMD) to ensure the learned mapping covers the target space uniformly.
- For discrete cases, they use KL divergence to ensure the model utilizes all available entries in a codebook rather than just a few.
- InfoInv (Information-Invariant) Transformation: The authors mathematically derive a specific transformation form that inherently preserves the relative spatial relationships between points. Interestingly, this derived form coincides with the positional encoding used in the original NeRF, providing a theoretical justification for why that heuristic works so well.
To make the discrete case trainable, they also utilize a differentiable Top-k operation with a reparameterization trick, allowing gradients to flow through the process of selecting codebook indices.
Datasets
The framework was evaluated across several standard benchmarks for 3D reconstruction and novel view synthesis:
- Synthetic-NeRF: The classic "Lego" and "Drums" dataset.
- S-NVSF & T&T (Tanks and Temples): Real-world and complex synthetic scenes.
- DTU: Used for evaluating both radiance fields and Signed Distance Fields (SDF) for surface reconstruction.
Key Insights and Ideas
- Information is the Anchor: The core "aha moment" is treating coordinate mapping as a communication problem. If the mapping (the gauge transformation) destroys information, the neural field can't recover the scene. Maximizing mutual information is the mathematically grounded way to prevent this.
- Geometry-Aware Regularization: Standard losses like KL divergence don't care about the "distance" between coordinates in a geometric space. By using Earth Mover's Distance for continuous mappings, the authors ensure the model respects the underlying geometry of the target space.
- Positional Encoding Revisited: Finding that their derived "Information-Invariant" mapping matches NeRF's positional encoding suggests that these encodings are essentially a hard-coded way to preserve spatial information during gauge transformation.
Key Results
- Superior Quality: In continuous mapping tasks (3D to 2D), the learned approach achieved a PSNR of 30.74, vastly outperforming the fixed orthogonal mapping baseline of 18.20.
- Effective Codebook Usage: For discrete mappings, InfoReg achieved near-perfect codebook utilization (1.000) and higher PSNR than standard spatial hashing.
- Versatility: The InfoInv transformation improved both grid-based and MLP-based neural fields, boosting both convergence speed and final rendering quality across the board.
- Qualitative Success: As shown in Figure 5, the learned mapping successfully captures complex textures in 360° scenes where fixed projections fail and result in blurred or missing details.
Limitations and Future Work
- Computational Overhead: While InfoInv is efficient, the full InfoReg regularization adds significant computational cost during training, making it currently unsuitable for "instant" training applications like Instant-NGP.
- Scaling Issues: For very large discrete codebooks (e.g., $2^{24}$ entries), the output layer of the gauge network becomes prohibitively large. The authors suggest grouping codebooks into sub-blocks as a potential future solution.
- Broader Applications: The authors see potential for applying these principles to other areas of discrete representation learning, such as VQ-VAEs, which also suffer from codebook collapse.
Summary updated