Revisiting Nonlocal Self-Similarity from Continuous Representation

Yisi Luo, Xile Zhao, Deyu Meng

TL;DR

This paper introduces the Continuous Representation-based NonLocal (CRNL) method, a novel approach that extends the powerful Nonlocal Self-Similarity (NSS) prior to handle both structured (on-meshgrid) and unstructured (off-meshgrid) data. CRNL achieves this by learning a continuous representation of data and employing a coupled low-rank function factorization to efficiently model self-similar groups, leading to improved performance and versatility across diverse data processing tasks.

What Problem Does This Paper Address?

The Nonlocal Self-Similarity (NSS) prior has been a cornerstone in various data processing tasks, particularly for recovering corrupted images and videos. NSS leverages the observation that similar local patterns often repeat throughout a signal. However, existing NSS-based methods are fundamentally limited to data with a regular grid structure (e.g., images, videos). This limitation excludes a growing class of important real-world data, such as point clouds and climate data, which are inherently unstructured or "off-meshgrid." The paper addresses this gap by proposing a unified framework that can effectively exploit NSS for both on-meshgrid and off-meshgrid data.

Methods

The core innovation of CRNL lies in its two key features:

  1. Continuous Representation: Instead of working with discrete patches, CRNL first learns a continuous representation of the observed data using an Implicit Neural Representation (INR). This INR, typically a neural network, maps coordinates to data values, allowing for a seamless representation of data regardless of its underlying structure.
  2. Coupled Low-Rank Function Factorization: After splitting the continuous space into "basic cubes" and grouping similar cubes, CRNL employs a novel coupled low-rank function factorization. This approach uses shared "factor functions" across different groups to capture inter-group similarity, while "unshared core tensors" preserve the individuality of each group. This contrasts with traditional methods that treat each group independently, thus missing out on valuable cross-group correlations. This factorization is designed to be computationally efficient and to better model the underlying data structures.

Datasets

The paper evaluates CRNL on a variety of datasets across different tasks:

  • Image Inpainting: Color images (Peppers, Plane, Sailboat, House), multispectral images (MSIs) (Toys, Cloth from CAVE dataset), and videos (Foreman, Carphone).
  • Image Denoising: Multispectral images (MSIs) (Cups, Fruits from CAVE dataset) and hyperspectral images (HSIs) (WDC-1, WDC-2, Pavia U-1, Pavia U-2).
  • Multivariate Regression:
    • Synthetic datasets generated from four explicit functions (f1-f4).
  • Climate data (precipitation data from four North American locations).
  • Point cloud data (four datasets: Mario, Rabbit, Frog, Duck).

Key Insights and Ideas

  • Bridging the Meshgrid Divide: The most significant insight is the successful generalization of the NSS prior to off-meshgrid data by leveraging continuous representations. This opens up NSS-based methods to a much wider range of real-world applications.
  • The Power of Shared Representations: The coupled low-rank function factorization is crucial. By sharing factor functions across groups, CRNL captures inter-group similarities that are missed by traditional methods, leading to more accurate modeling and better performance.
  • Efficiency Through Compactness: The coupled factorization not only improves accuracy but also significantly enhances computational efficiency by reducing the number of parameters needed to represent multiple groups.
  • INR as a Foundation: The use of INR as the initial step provides a flexible and powerful way to represent diverse data types, making the subsequent NSS processing more robust.

Key Results

CRNL demonstrates state-of-the-art or competitive performance across all tested tasks:

  • Image Inpainting: CRNL consistently achieves the best quantitative results (PSNR, SSIM) and visually recovers finer details compared to other unsupervised methods. It also shows favorable computational efficiency.
  • Image Denoising: CRNL achieves satisfactory quantitative results, outperforming other methods, particularly in recovering details and removing noise. Its running time is also efficient.
  • Multivariate Regression: CRNL exhibits higher accuracy than standard regression methods and specialized tensor decomposition methods on synthetic, climate, and point cloud data. It effectively recovers complex structures, especially in climate data.
  • Computational Efficiency: The paper highlights that CRNL (coupled) is significantly faster than its uncoupled counterpart and other NSS-based methods, often by a factor of three or more, due to the shared factor functions.

Limitations and Future Work

  • Hyperparameter Sensitivity: While the paper suggests CRNL is relatively robust to hyperparameters (rank, cube size, number of similar cubes), tuning them can still impact performance. Future work could explore more automated hyperparameter optimization.
  • Iterative Regularization: The authors suggest enhancing CRNL's capabilities by incorporating techniques like iterative regularization and weighted schemes.
  • Broader Applications: The paper points to potential future applications in areas like hyperspectral fusion and spherical image processing, indicating a broad applicability of the proposed framework.

Summary updated

Abstract

Nonlocal self-similarity (NSS) is an important prior that has been successfully applied in multi-dimensional data processing tasks, e.g., image and video recovery. However, existing NSS-based methods are solely suitable for meshgrid data such as images and videos, but are not suitable for emerging off-meshgrid data, e.g., point cloud and climate data. In this work, we revisit the NSS from the continuous representation perspective and propose a novel Continuous Representation-based NonLocal method (termed as CRNL), which has two innovative features as compared with classical nonlocal methods. First, based on the continuous representation, our CRNL unifies the measure of self-similarity for on-meshgrid and off-meshgrid data and thus is naturally suitable for both of them. Second, the nonlocal continuous groups can be more compactly and efficiently represented by the coupled low-rank function factorization, which simultaneously exploits the similarity within each group and across different groups, while classical nonlocal methods neglect the similarity across groups. This elaborately designed coupled mechanism allows our method to enjoy favorable performance over conventional NSS methods in terms of both effectiveness and efficiency. Extensive multi-dimensional data processing experiments on-meshgrid (e.g., image inpainting and image denoising) and off-meshgrid (e.g., climate data prediction and point cloud recovery) validate the versatility, effectiveness, and efficiency of our CRNL as compared with state-of-the-art methods.
BibTeX
@article{luo2025revisiting,
  author = {Yisi Luo and Xile Zhao and Deyu Meng},
  title = {{Revisiting Nonlocal Self-Similarity from Continuous Representation}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2025},
  doi = {10.1109/tpami.2024.3464875},
}
Figure