General Neural Gauge Fields

Fangneng Zhan, Lingjie Liu, Adam Kortylewski, Christian Theobalt

TL;DR

This paper introduces General Neural Gauge Fields, a framework that allows neural scene representations to learn their own optimal coordinate systems (gauge transformations) instead of relying on fixed, pre-defined ones. By deriving new regularization techniques and invariant functions from the principle of information conservation, the authors prevent the common problem of "gauge collapse" and achieve significantly better rendering quality and efficiency.

What Problem Does This Paper Address?

Modern 3D scene representations, like Neural Radiance Fields (NeRF), often don't work directly with raw 3D coordinates. To speed up training or save memory, researchers use "gauge transformations"—mappings that project 3D points into other systems like 2D tri-planes or discrete hash tables.

However, these mappings are almost always fixed heuristics (e.g., simple orthogonal projections). While they work, they are sub-optimal because they don't adapt to the specific geometry of a scene. The obvious solution is to learn the mapping alongside the scene itself. The catch? Naively learning these transformations leads to gauge collapse: the model finds a "lazy" solution where it maps many different 3D points to the same target coordinate, effectively losing the information needed to reconstruct the scene.

Methods

The authors propose a unified framework for learning both continuous (e.g., mapping to a 2D plane) and discrete (e.g., mapping to a codebook of vectors) gauge transformations. To solve the collapse problem, they introduce two key innovations:

  1. InfoReg (Information Regularization): Derived from the principle of information conservation, this loss function maximizes the mutual information between the original 3D points and the new target coordinates.
  2. For continuous cases, they use Earth Mover’s Distance (EMD) to ensure the learned mapping covers the target space uniformly.
  3. For discrete cases, they use KL divergence to ensure the model utilizes all available entries in a codebook rather than just a few.
  4. InfoInv (Information-Invariant) Transformation: The authors mathematically derive a specific transformation form that inherently preserves the relative spatial relationships between points. Interestingly, this derived form coincides with the positional encoding used in the original NeRF, providing a theoretical justification for why that heuristic works so well.

To make the discrete case trainable, they also utilize a differentiable Top-k operation with a reparameterization trick, allowing gradients to flow through the process of selecting codebook indices.

Conceptual illustration of learning a mapping from 3D points to a 2D plane compared to fixed projections.

Datasets

The framework was evaluated across several standard benchmarks for 3D reconstruction and novel view synthesis:

  • Synthetic-NeRF: The classic "Lego" and "Drums" dataset.
  • S-NVSF & T&T (Tanks and Temples): Real-world and complex synthetic scenes.
  • DTU: Used for evaluating both radiance fields and Signed Distance Fields (SDF) for surface reconstruction.

Key Insights and Ideas

  • Information is the Anchor: The core "aha moment" is treating coordinate mapping as a communication problem. If the mapping (the gauge transformation) destroys information, the neural field can't recover the scene. Maximizing mutual information is the mathematically grounded way to prevent this.
  • Geometry-Aware Regularization: Standard losses like KL divergence don't care about the "distance" between coordinates in a geometric space. By using Earth Mover's Distance for continuous mappings, the authors ensure the model respects the underlying geometry of the target space.
  • Positional Encoding Revisited: Finding that their derived "Information-Invariant" mapping matches NeRF's positional encoding suggests that these encodings are essentially a hard-coded way to preserve spatial information during gauge transformation.
Comparison showing how InfoReg prevents continuous gauge collapse compared to other regularization methods.

Key Results

  • Superior Quality: In continuous mapping tasks (3D to 2D), the learned approach achieved a PSNR of 30.74, vastly outperforming the fixed orthogonal mapping baseline of 18.20.
  • Effective Codebook Usage: For discrete mappings, InfoReg achieved near-perfect codebook utilization (1.000) and higher PSNR than standard spatial hashing.
  • Versatility: The InfoInv transformation improved both grid-based and MLP-based neural fields, boosting both convergence speed and final rendering quality across the board.
  • Qualitative Success: As shown in Figure 5, the learned mapping successfully captures complex textures in 360° scenes where fixed projections fail and result in blurred or missing details.
Qualitative results showing learned mappings capturing scene textures better than fixed orthogonal mappings.

Limitations and Future Work

  • Computational Overhead: While InfoInv is efficient, the full InfoReg regularization adds significant computational cost during training, making it currently unsuitable for "instant" training applications like Instant-NGP.
  • Scaling Issues: For very large discrete codebooks (e.g., $2^{24}$ entries), the output layer of the gauge network becomes prohibitively large. The authors suggest grouping codebooks into sub-blocks as a potential future solution.
  • Broader Applications: The authors see potential for applying these principles to other areas of discrete representation learning, such as VQ-VAEs, which also suffer from codebook collapse.

Summary updated

Abstract

The recent advance of neural fields, such as neural radiance fields, has significantly pushed the boundary of scene representation learning. Aiming to boost the computation efficiency and rendering quality of 3D scenes, a popular line of research maps the 3D coordinate system to another measuring system, e.g., 2D manifolds and hash tables, for modeling neural fields. The conversion of coordinate systems can be typically dubbed as \emph{gauge transformation}, which is usually a pre-defined mapping function, e.g., orthogonal projection or spatial hash function. This begs a question: can we directly learn a desired gauge transformation along with the neural field in an end-to-end manner? In this work, we extend this problem to a general paradigm with a taxonomy of discrete \& continuous cases, and develop a learning framework to jointly optimize gauge transformations and neural fields. To counter the problem that the learning of gauge transformations can collapse easily, we derive a general regularization mechanism from the principle of information conservation during the gauge transformation. To circumvent the high computation cost in gauge learning with regularization, we directly derive an information-invariant gauge transformation which allows to preserve scene information inherently and yield superior performance. Project: https://fnzhan.com/Neural-Gauge-Fields
BibTeX
@inproceedings{zhan2023general,
  author = {Fangneng Zhan and Lingjie Liu and Adam Kortylewski and Christian Theobalt},
  title = {{General Neural Gauge Fields}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=XWkWK2UagFR},
}
Figure