General Neural Gauge Fields

Fangneng Zhan, Lingjie Liu, Adam Kortylewski, Christian Theobalt

TL;DR

Neural radiance fields (NeRFs) often rely on fixed, "hard-coded" coordinate mappings—like projecting 3D points onto 2D planes—to achieve high-speed rendering. This paper introduces Neural Gauge Fields, a framework that allows these mappings to be learned end-to-end alongside the scene itself. By applying a novel information-theoretic regularization (InfoReg) to prevent the mapping from collapsing, the authors achieve significantly better reconstruction quality and more flexible scene representations.

The Problem: Fixed vs. Learned Mappings

Representing 3D scenes with neural networks is powerful but computationally expensive. To speed things up, modern "explicit" methods map 3D coordinates into other systems, such as 2D "tri-planes" (EG3D) or hash tables (Instant-NGP). In physics and geometry, these coordinate systems are called gauges, and moving between them is a gauge transformation.

Currently, most researchers use pre-defined, fixed gauge transformations (e.g., simple orthogonal projections). The authors argue that these fixed mappings are sub-optimal because they don't adapt to the specific geometry of a scene. However, simply trying to "learn" the mapping usually fails due to gauge collapse: the network finds a "shortcut" where it maps all 3D points to a single tiny spot in the new coordinate system, destroying all scene information.

Proposed Solution: Neural Gauge Fields

The authors propose a unified framework for learning both continuous (mapping 3D to 2D planes) and discrete (mapping 3D to codebook indices) gauge transformations. To make this work, they introduce two key innovations:

  1. InfoReg (Information Regularization): Derived from the principle of information conservation, this loss function maximizes the mutual information between the original 3D points and the transformed coordinates.
  2. For continuous cases, they use Earth Mover’s Distance (EMD) to ensure the learned mapping spreads points uniformly across the target 2D plane.
  3. For discrete cases, they use KL divergence to ensure the model utilizes the entire available codebook rather than just a few entries.
  4. InfoInv (Information-Invariant Transformation): Learning with InfoReg can be computationally heavy. To solve this, the authors mathematically derive a specific transformation function that inherently preserves scene information. Interestingly, this derived form coincides with the "positional encoding" used in the original NeRF, providing a new theoretical justification for why that technique is so effective.

Key Insights and Ideas

  • Learned vs. Fixed Mappings: A learned mapping can "unwrap" a complex 3D object onto a 2D plane much more efficiently than a fixed projection. Think of the difference between a standard map projection and a custom "UV unwrap" used by 3D artists—the latter preserves much more detail.
Comparison of fixed orthogonal mappings versus a learned mapping for a 3D Swiss roll structure.
  • Information as a Constraint: The core breakthrough is treating coordinate transformation as an information bottleneck. By forcing the network to conserve information, it naturally prevents the mathematical "collapse" that usually plagues learned mappings.
  • The Top-k Trick: In discrete mappings (like hash tables), the authors found that selecting the "Top-k" most probable indices (rather than just the single best one) significantly increases the information capacity and helps alleviate collapse.
  • Re-evaluating Positional Encoding: By showing that NeRF’s positional encoding is actually an information-invariant gauge transformation, the paper bridges the gap between heuristic engineering and fundamental geometric theory.

Key Results

The authors evaluated their approach on several standard 3D reconstruction benchmarks, including Synthetic-NeRF, Synthetic-NSVF, DTU, and Tanks and Temples.

  • Massive Quality Gains: In continuous mapping tasks (3D to 2D), the learned approach achieved a 30.74 PSNR on the S-NeRF dataset, compared to just 18.20 PSNR for standard orthogonal projections.
  • Superior Regularization: InfoReg consistently outperformed previous "heuristic" methods like Cycle Consistency or Structural Regularization. In discrete cases, InfoReg achieved nearly perfect codebook utilization (1.000) where other methods failed.
  • Boosting SOTA: When added to existing state-of-the-art models like TensoRF or Instant-NGP, the InfoInv transformation provided a "free" performance boost (e.g., increasing TensoRF's PSNR from 32.32 to 32.96) with negligible increase in training time or memory.

Limitations and Future Work

  • Scalability: While the method works well for moderate codebook sizes, applying it to the massive hash tables used in Instant-NGP (which can have millions of entries) is challenging because the "gauge network" required to predict those indices becomes prohibitively large.
  • Computational Cost: While InfoInv is fast, the full InfoReg (especially the EMD calculation) adds significant overhead to the training process.
  • Future Directions: The authors suggest extending this framework to dynamic scenes (where the gauge might change over time) and surface reconstruction tasks using Signed Distance Fields (SDFs).

Summary updated

Abstract

The recent advance of neural fields, such as neural radiance fields, has significantly pushed the boundary of scene representation learning. Aiming to boost the computation efficiency and rendering quality of 3D scenes, a popular line of research maps the 3D coordinate system to another measuring system, e.g., 2D manifolds and hash tables, for modeling neural fields. The conversion of coordinate systems can be typically dubbed as \emph{gauge transformation}, which is usually a pre-defined mapping function, e.g., orthogonal projection or spatial hash function. This begs a question: can we directly learn a desired gauge transformation along with the neural field in an end-to-end manner? In this work, we extend this problem to a general paradigm with a taxonomy of discrete and continuous cases, and develop an end-to-end learning framework to jointly optimize the gauge transformation and neural fields. To counter the problem that the learning of gauge transformations can collapse easily, we derive a general regularization mechanism from the principle of information conservation during the gauge transformation. To circumvent the high computation cost in gauge learning with regularization, we directly derive an information-invariant gauge transformation which allows to preserve scene information inherently and yield superior performance.
BibTeX
@inproceedings{zhan2023general,
  author = {Fangneng Zhan and Lingjie Liu and Adam Kortylewski and Christian Theobalt},
  title = {{General Neural Gauge Fields}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=XWkWK2UagFR},
}
Figure