Low-Rank Adaptation of Neural Fields

Anh Truong, Ahmed H. Mahmoud, Mina Konakovic-Lukovic, Justin Solomon

Introduction

Neural fields—typically implemented as Multi-Layer Perceptrons (MLPs) that map coordinates to values like color, density, or distance—have revolutionized representations in computer graphics. However, editing these fields remains a significant challenge. Traditional methods require full fine-tuning of the network weights to incorporate changes, such as a new frame in a video or a deformation in a 3D mesh. This approach is highly redundant and storage-intensive, as it requires saving a complete set of weights for every minor variation.

This paper introduces Low-Rank Adaptation (LoRA) to the domain of neural fields. By representing edits as compact, low-rank updates to a base model's weights, the authors achieve high-fidelity results while using significantly fewer parameters than traditional fine-tuning.

Methodology: Adapting the Delta

The core idea is to keep the base weight matrix $W$ of a pre-trained neural field frozen. Instead of updating $W$ directly, the method learns an additive update $\Delta W$. To ensure parameter efficiency, $\Delta W$ is constrained to be low-rank through factorization into two smaller matrices, $A$ and $B$, such that $\Delta W = BA$. For a weight matrix of size $d \times k$, the rank $r$ is chosen such that $r \ll \min(d, k)$, drastically reducing the number of trainable parameters.

LoRA enables efficient updates across diverse modalities including 3D deformations, image edits, and temporal sequences.

Key Implementation Details

  • Instance-Specific Adaptation: Unlike its use in Large Language Models (LLMs) for task-switching, LoRA is applied here to instance-specific models (e.g., an MLP overfit to a single 3D mesh).
  • Sequential Updates: For temporal data like video, LoRA adapters are applied sequentially. Each frame's update builds upon the previous one, allowing the model to track long-term changes without accumulating significant error drift.
  • Optimization Loops: The authors demonstrate that LoRA parameters can be optimized directly within energy minimization frameworks, such as denoising an image by minimizing Total Variation (TV) loss.

Empirical Evaluation

The researchers evaluated the method across four primary modalities: high-resolution images (e.g., depth-of-field shifts), 3D geometry (SDFs of characters like a Penguin or Armadillo), physical simulations (Jellyfish), and video (1080p traffic footage).

Key Insights

  1. Edits are Inherently Low-Rank: The research empirically confirms that common graphics edits—filtering, local deformations, and lighting changes—correspond to very low-rank shifts in the neural network's weight space.
  2. Learning vs. Compression: A critical finding is that learning a low-rank update via LoRA is far superior to taking a full fine-tuned update and compressing it using Singular Value Decomposition (SVD) after the fact. LoRA consistently outperformed SVD-based compression by over 10 dB in PSNR.
  3. Efficiency and Fidelity: At a rank of $r=16$, which represents roughly 12–15% of the original parameters, the reconstructions are often visually indistinguishable from full fine-tuning.
Error decay as a function of rank for different image datasets, showing that a small rank is sufficient to capture the delta across all network layers.

Results and Performance

  • Parameter Savings: The method achieves high-fidelity results with 7–8× fewer parameters than full fine-tuning.
  • Quantitative Gains: In image tasks, LoRA outperformed "Small MLP" baselines (training a smaller network from scratch with the same parameter budget) by 1.9 dB, proving the structural advantage of the "base + delta" approach.
  • Geometric Precision: For 3D shapes, LoRA maintained an Intersection-over-Union (IoU) of >0.99, preserving intricate surface details that smaller networks failed to capture.

Limitations and Future Work

While LoRA provides massive storage savings, it does not currently offer a training speedup, as backpropagation still requires passes through the full base model. Additionally, the method faces challenges with "global" high-frequency edits, such as complex stylization filters that affect the entire image uniformly. Future research may explore extending this technique to more complex architectures like Hash-grids or Tri-planes to enable real-time neural rendering and editing.

Summary updated

BibTeX
@inproceedings{truong2025low,
  author = {Anh Truong and Ahmed H. Mahmoud and Mina Konakovic-Lukovic and Justin Solomon},
  title = {{Low-Rank Adaptation of Neural Fields}},
  booktitle = {SIGGRAPH Asia},
  year = {2025},
  doi = {10.1145/3757377.3763882},
}
Figure