Implicit-Zoo: A Large-Scale Dataset of Neural Implicit Functions for 2D Images and 3D Scenes

Qi Ma, Danda Pani Paudel, Ender Konukoglu, Luc Van Gool

Introduction

Neural Implicit Representations (INRs) have emerged as a powerful alternative to traditional discrete data formats like pixels or voxels. By representing data as continuous functions—typically Multi-Layer Perceptrons (MLPs) that map coordinates to values (e.g., $(x, y) \to RGB$)—INRs offer high fidelity, memory efficiency, and resolution independence. However, the field has faced two significant bottlenecks: the massive computational cost of training high-quality INRs from scratch and the lack of a large-scale, standardized dataset for generalizable research.

Implicit-Zoo addresses these hurdles by providing a massive open-source library of over 1.5 million pre-trained INRs, effectively serving as the "ImageNet" for the implicit representation community.

The Implicit-Zoo Dataset

The dataset is designed to support a wide range of computer vision tasks across 2D and 3D modalities. The authors utilized SIREN (Sinusoidal Representation Networks) for 2D data and NeRF (Neural Radiance Fields) for 3D objects. To ensure high fidelity, a strict three-phase quality control process was implemented, requiring all 2D INRs to meet a minimum Peak Signal-to-Noise Ratio (PSNR) of 30dB.

Dataset Composition:

  • 2D Tasks:
    • ImageNet-1K-INRs: Over 1.4 million scenes.
  • CIFAR-10-INRs: 60,000 scenes.
  • Cityscapes-INRs: 23,473 scenes for semantic segmentation.
  • 3D Tasks:
    • OmniObject3D-INRs: 5,914 3D objects spanning 190 categories.

Methodology: Learnable Tokenization and Pose Regression

Beyond the dataset itself, the researchers introduced novel techniques to optimize how models interact with continuous representations.

Learnable Tokenization for Transformers

Standard Vision Transformers (ViTs) divide images into a rigid, fixed grid of patches. Because INRs are continuous, Implicit-Zoo introduces a Learnable Tokenizer. This allows the network to learn the optimal coordinates, scales, and centers for its sampling patches. By optimizing where the model "looks," the system can focus on more informative regions of a scene. To prevent the sampling points from collapsing into local minima (clumping together), the authors introduced a regularization term that penalizes coordinates that become too close, maintaining a flexible but effective distribution.

Transformer-based Pose Regressor

For 3D applications, the paper proposes a method to extract volume features directly from 3D INRs. These features are integrated with 2D image features to regress camera poses directly. This approach is significant because it bypasses the need for the coarse initialization required by many existing pose estimation methods, paving the way for more robust robot re-localization and AR/VR applications.

Experimental Results

The authors benchmarked Implicit-Zoo across three primary tasks, demonstrating the advantages of learnable sampling over fixed grids:

  1. Classification: On CIFAR-10-INRs, the learnable tokenizer (LP+Reg) achieved 81.57% accuracy, outperforming the standard ViT baseline of 80.82%.
  2. Segmentation: On the Cityscapes dataset, the learnable tokenizer improved the mean Intersection over Union (mIOU) from 39.95 to 40.61 for fine annotations.
  3. 3D Pose Regression: In unseen scenarios, the model achieved a rotational error of 20°, with nearly 80% of poses falling below a 30° error threshold, establishing a strong baseline for future implicit 3D research.

Discussion and Future Work

While Implicit-Zoo represents a major leap forward, several challenges remain. Query Latency is a primary concern, as running an MLP forward pass is computationally more expensive than reading a pixel from a buffer. Additionally, while the 30dB PSNR threshold is high, some visual artifacts persist in repetitive backgrounds or small objects. Finally, the 3D pose regressor still faces difficulties with perfectly symmetric objects, where different camera angles produce identical visual outputs.

Future research will likely focus on expanding the dataset to more diverse modalities and optimizing the inference speed of INR-based architectures to make them more competitive with traditional discrete models.

Data and Code Availability: https://github.com/qimaqi/Implicit-Zoo/

Summary updated

BibTeX
@inproceedings{ma2024implicit,
  author = {Qi Ma and Danda Pani Paudel and Ender Konukoglu and Luc Van Gool},
  title = {{Implicit-Zoo: A Large-Scale Dataset of Neural Implicit Functions for 2D Images and 3D Scenes}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=j2yphY2779},
}
Figure