NeurTV: Total Variation on the Neural Domain

Yisi Luo, Xile Zhao, Kai Ye, Deyu Meng

TL;DR

NeurTV redefines the classic Total Variation (TV) regularization by shifting it from the discrete pixel domain to a continuous "neural domain." By representing data through a differentiable deep neural network, the authors enable TV-based signal recovery that is free of discretization errors and natively supports non-grid data like point clouds and spatial transcriptomics.

What Problem Does This Paper Address?

Total Variation (TV) has been a cornerstone of image processing for decades, prized for its ability to recover sharp edges while removing noise. However, traditional TV has two major "Achilles' heels":

  1. Grid Dependency: It relies on discrete differences between adjacent pixels. This makes it difficult to apply to unstructured data, such as 3D point clouds or biological spatial transcriptomics, where there is no fixed "neighbor" grid.
  2. Discretization Error: The discrete difference operator is only an approximation of a true derivative. This leads to artifacts like the "staircase effect" and limits the precision of directional smoothing.

The authors aim to break these limitations by moving TV into a continuous space where derivatives can be calculated exactly using the machinery of deep learning.

Methods

The core idea is to use a Continuous Representation. Instead of storing an image as a matrix of pixels, the authors represent it as a function $f_\Theta(\mathbf{x})$—a deep neural network (DNN) that takes a coordinate $\mathbf{x}$ as input and outputs the signal value (e.g., color).

NeurTV then defines regularization not by subtracting pixel values, but by penalizing the partial derivatives of the network's output with respect to its input coordinates: $$\Psi_{\text{NeurTV}}(\Theta) := \sum_{d=1}^N \sum_{\mathbf{x} \in \Gamma} \left| \frac{\partial f_\Theta(\mathbf{x})}{\partial x_{(d)}} \right|$$

Key methodological highlights include:

  • Backbone Flexibility: The framework works with various coordinate-based architectures, including SIREN (sine-based), Positional Encoding (PE), and Tensor Factorization (TF) networks.
  • Space-Variant NeurTV: The authors introduce adaptive weights that change based on local data structures, allowing for stronger smoothing in flat regions and better edge preservation in textured areas.
  • Higher-Order & Directional Variants: Because the DNN is differentiable, calculating second-order derivatives or derivatives along arbitrary angles becomes straightforward, avoiding the complex discrete operators required in traditional methods.
The authors test NeurTV with various DNN architectures, including sine-based and tensor factorization models.

Datasets

The authors evaluated NeurTV across a diverse set of data modalities:

  • Color Images: Standard benchmarks like "Peppers," "Boat," and "House."
  • Hyperspectral Images (HSI): Datasets including WDC, Pavia, and the CAVE dataset.
  • Point Clouds: 3D models such as "Mario," "Duck," and "Rabbit."
  • Spatial Transcriptomics: A mouse brain dataset used to reconstruct gene expression patterns from sparse samples.

Key Insights and Ideas

  • The "Neural Domain" Advantage: By treating data as a continuous function, NeurTV becomes "resolution-independent." You can sample the regularizer at a higher density than the original data to get a more accurate approximation of the true functional TV.
  • Native Support for Unstructured Data: Since the method only requires coordinates and signal values, it treats a point cloud exactly like an image. Traditional TV simply cannot be applied to point clouds without first projecting them onto a grid, which loses information.
  • Automatic Differentiation: Leveraging modern deep learning libraries (like PyTorch's autograd) allows for exact derivative calculations, eliminating the discretization errors inherent in pixel-based differences.
NeurTV handles both standard images and unstructured point clouds, where traditional TV is "not capable."

Key Results

NeurTV consistently outperformed both classical TV and state-of-the-art deep learning baselines:

  • Image Denoising: On the "Peppers" image with Gaussian noise, NeurTV achieved a PSNR of 25.93, significantly higher than classical TV (23.13) and directional TV methods.
  • Point Cloud Recovery: NeurTV successfully reconstructed 3D color information for point clouds with high accuracy (MSE of 0.090 for "Mario"), whereas traditional TV methods were inapplicable.
  • Staircase Effect Reduction: The second-order variant of NeurTV effectively mitigated the blocky artifacts common in first-order TV methods, producing much smoother gradients in recovered images.
Quantitative results for point cloud recovery show NeurTV outperforming SIREN and other regression baselines.

Limitations and Future Work

  • Hyperparameter Tuning: The trade-off parameter $\lambda$ (which balances data fidelity vs. smoothing) currently requires manual tuning. The authors suggest that learning this parameter automatically through the network is a promising future direction.
  • Computational Cost: Training a coordinate-based neural network for every signal recovery task is generally slower than applying a simple discrete filter, though the authors note that the results justify the cost for high-stakes applications.
  • 3D Directional Extensions: While directional NeurTV was demonstrated for 2D images, extending it to 3D point clouds is noted as a non-trivial task left for future research.

Summary updated

Abstract

Recently, we have witnessed the success of total variation (TV) for many imaging applications. However, traditional TV is defined on the original pixel domain, which limits its potential. In this work, we suggest a new TV regularization defined on the neural domain. Concretely, the discrete data is implicitly and continuously represented by a deep neural network (DNN), and we use the derivatives of DNN outputs w.r.t. input coordinates to capture local correlations of data. As compared with classical TV on the original domain, the proposed TV on the neural domain (termed NeurTV) enjoys the following advantages. First, NeurTV is free of discretization error induced by the discrete difference operator. Second, NeurTV is not limited to meshgrid but is suitable for both meshgrid and non-meshgrid data. Third, NeurTV can more exactly capture local correlations across data for any direction and any order of derivatives attributed to the implicit and continuous nature of neural domain. We theoretically reinterpret NeurTV under the variational approximation framework, which allows us to build the connection between NeurTV and classical TV and inspires us to develop variants (e.g., space-variant NeurTV). Extensive numerical experiments with meshgrid data (e.g., color and hyperspectral images) and non-meshgrid data (e.g., point clouds and spatial transcriptomics) showcase the effectiveness of the proposed methods.
BibTeX
@article{luo2025neurtv,
  author = {Yisi Luo and Xile Zhao and Kai Ye and Deyu Meng},
  title = {{NeurTV: Total Variation on the Neural Domain}},
  journal = {SIAM J. Imaging Sci.},
  year = {2025},
  doi = {10.1137/24m1664605},
}
Figure