NeurTV: Total Variation on the Neural Domain

Yisi Luo, Xile Zhao, Kai Ye, Deyu Meng

TL;DR

This paper introduces NeurTV, a novel regularization technique that extends traditional Total Variation (TV) by defining it on the continuous "neural domain" learned by a Deep Neural Network (DNN). This allows NeurTV to overcome limitations of classical TV, enabling it to handle non-meshgrid data and capture local correlations more precisely across any direction and order of derivatives, leading to improved performance in various imaging tasks.

What Problem Does This Paper Address?

Traditional Total Variation (TV) regularization is a powerful tool for image processing tasks like denoising and inpainting. It works by penalizing the differences between adjacent pixels, effectively promoting piecewise constant or smooth solutions. However, classical TV has two major limitations:

  1. Meshgrid Dependency: It's inherently designed for data arranged on a grid (like images) and struggles with unstructured or non-meshgrid data (e.g., point clouds, spatial transcriptomics).
  2. Limited Correlation Capture: It primarily considers local correlations in specific directions (horizontal and vertical) and struggles to capture more complex, multi-directional, or higher-order local relationships within the data.

These limitations restrict TV's applicability and effectiveness in modern imaging scenarios dealing with diverse data types and complex structures.

Methods

The core innovation is NeurTV, which redefines TV on a "neural domain." Instead of operating on discrete pixel differences, NeurTV leverages a Deep Neural Network (DNN) to learn a continuous representation of the data.

Here's how it works:

  • Continuous Representation: A DNN, denoted as $f_\Theta(x)$, is trained to map input coordinates $x$ to the data value. This allows the data to be represented implicitly and continuously, rather than as a discrete grid.
  • Neural Domain Derivatives: NeurTV uses the derivatives of the DNN's output ($f_\Theta(x)$) with respect to its input coordinates ($x$) to capture local correlations. This is analogous to how classical TV uses pixel differences.
  • Advantages:
    • Meshgrid Agnostic: Because it operates on the continuous output of a DNN, NeurTV is naturally applicable to both meshgrid (images) and non-meshgrid (point clouds, spatial transcriptomics) data.
  • Enhanced Correlation Capture: By using derivatives, NeurTV can capture local correlations in any direction and of any order, offering a more comprehensive understanding of data structure.
  • No Discretization Error: Unlike classical TV which approximates derivatives with discrete differences, NeurTV uses direct derivatives from the DNN, avoiding discretization errors.
  • Variational Approximation: The authors theoretically re-interpret NeurTV through a variational approximation framework. This not only connects NeurTV to classical TV but also inspires new variants like arbitrary resolution NeurTV and space-variant NeurTV, which adapt parameters spatially.

Datasets

The paper evaluates NeurTV on a variety of datasets across different applications:

  • Image Denoising/Inpainting: Standard image datasets like "Peppers," "Boat," "House," "Plane," and "Earth" are used.
  • Hyperspectral Image (HSI) Mixed Noise Removal: Datasets like "WDC," "Pavia," and HSIs from the CAVE dataset ("Cloth," "Cups," "Fruits").
  • Point Cloud Recovery: Datasets include "Cola," "Mario," "Duck," "Squirrel," and "Rabbit."
  • Spatial Transcriptomics Reconstruction: Data from a mouse brain dataset, focusing on gene types like "mbp," "snap25," "atp1b1," "plp1," and "rtn1."

No new datasets were introduced; existing benchmark datasets were used to demonstrate NeurTV's versatility.

Key Insights and Ideas

  • Bridging Discrete and Continuous: The most significant insight is the successful translation of a discrete regularization concept (Total Variation) into a continuous domain using DNNs. This opens up new avenues for regularization techniques.
  • DNN as a Continuous Regularizer: The paper demonstrates that a DNN can serve not just as a powerful feature extractor but also as a foundation for learning complex regularization priors, going beyond simple data fitting.
  • Flexibility for Diverse Data: The meshgrid-agnostic nature of NeurTV is a major breakthrough, allowing a single framework to address problems across vastly different data modalities.
  • Theoretical Foundation for Practical Gains: The variational approximation reinterpretation provides a strong theoretical grounding for NeurTV's effectiveness and guides the development of its advanced variants.
  • Beyond First-Order Smoothness: The ability to easily extend NeurTV to higher-order and directional derivatives allows for capturing more intricate data structures, which is crucial for complex imaging tasks.

Key Results

NeurTV consistently outperforms classical TV and other state-of-the-art methods across various applications:

  • Image Denoising: NeurTV achieves higher PSNR values compared to classical TV, DIPWTV, and HDTV. For example, on "Peppers" with noise deviation 0.1, NeurTV reaches a PSNR of 25.93, outperforming HDTV (25.47).
  • Image Inpainting: NeurTV shows better performance than methods like STTC, HLRTF, and TCTV, achieving higher PSNR and SSIM scores. For instance, on "Plane" with sampling rate 0.2, NeurTV achieves a PSNR of 26.13.
  • Hyperspectral Image Mixed Noise Removal: NeurSSTV (a variant of NeurTV) outperforms LRTDTV, LRTF-DFR, and RCTV, achieving higher PSNR and SSIM. On "Fruits" with Gaussian noise 0.2 and impulse noise 0.1, NeurSSTV reaches a PSNR of 41.18.
  • Point Cloud Recovery: NeurTV consistently achieves lower MSE and higher R-Square values than KNR, DT, RF, FSA-HTF, and SIREN. For example, on "Mario" with sampling rate 0.2, NeurTV achieves an MSE of 0.079.
  • Spatial Transcriptomics Reconstruction: NeurTV outperforms GNTD and other methods, demonstrating its ability to capture complex spatial gene expression patterns.

Limitations and Future Work

  • Hyperparameter Tuning: The trade-off parameter $\lambda$ and the number of partitions $n$ require careful tuning for optimal performance, although the paper shows robustness within a suitable range.
  • Computational Cost: While effective, training DNNs can be computationally intensive.
  • Future Work:
    • Combining NeurTV with graph-based methods for further performance boosts.
  • Exploring automatic training of the trade-off parameter through the DNN itself.
  • Developing more sophisticated space-variant directional parameter selection strategies.
  • Investigating the combination of NeurTV with other regularization techniques.

Summary updated

Abstract

Recently, we have witnessed the success of total variation (TV) for many imaging applications. However, traditional TV is defined on the original pixel domain, which limits its potential. In this work, we suggest a new TV regularization defined on the neural domain. Concretely, the discrete data is continuously and implicitly represented by a deep neural network (DNN), and we use the derivatives of DNN outputs w.r.t. input coordinates to capture local correlations of data. As compared with classical TV on the original domain, the proposed TV on the neural domain (termed NeurTV) enjoys two advantages. First, NeurTV is not limited to meshgrid but is suitable for both meshgrid and non-meshgrid data. Second, NeurTV can more exactly capture local correlations across data for any direction and any order of derivatives attributed to the implicit and continuous nature of neural domain. We theoretically reinterpret NeurTV under the variational approximation framework, which allows us to build the connection between classical TV and NeurTV and inspires us to develop variants (e.g., NeurTV with arbitrary resolution and space-variant NeurTV). Extensive numerical experiments with meshgrid data (e.g., color and hyperspectral images) and non-meshgrid data (e.g., point clouds and spatial transcriptomics) showcase the effectiveness of the proposed methods.
BibTeX
@article{luo2025neurtv,
  author = {Yisi Luo and Xile Zhao and Kai Ye and Deyu Meng},
  title = {{NeurTV: Total Variation on the Neural Domain}},
  journal = {SIAM J. Imaging Sci.},
  year = {2025},
  doi = {10.1137/24m1664605},
}
Figure