The FineView Dataset:A 3D Scanned Multi-View Object Dataset of Fine-Grained Category Instances

Suguru Onda, Ryan FarrellWACV 2025

TL;DR

This paper introduces the FineView Dataset, a novel 3D-scanned, multi-view dataset of fine-grained insect categories with accurate camera calibration. This dataset addresses the limitations of existing datasets that often lack multi-view information and precise camera parameters, which are crucial for advancing fine-grained visual categorization (FGVC) and 3D representation learning.

What Problem Does This Paper Address?

Fine-grained visual categorization (FGVC) is a challenging task that involves distinguishing between highly similar object classes, such as different species of butterflies. While deep learning has made significant strides in FGVC using large image collections, these datasets often consist of web-scraped images that lack essential information like multiple views of the same object, 3D geometric data, or accurate camera parameters. This absence of rich, structured data hinders the development of models that can truly understand object shape, pose, and texture, much like humans do. Existing 3D datasets often lack fine-grained categories, and while some multi-view datasets exist, they typically don't offer precise camera calibration or focus on fine-grained distinctions. This paper aims to bridge this gap by providing a dataset that offers both multi-view imagery and 3D information with meticulous camera calibration, specifically targeting fine-grained categories.

Methods

The authors developed a custom bi-directional, multi-camera, and 3D scanning system to collect the FineView dataset. This system utilizes 8 DSLR cameras mounted on aluminum rails, along with projectors for structured-light illumination and softbox lights for standard RGB capture. The core of the data collection involves two modes: capturing high-resolution RGB images and capturing structured-light patterns. The latter uses unique markers projected onto the object's surface, allowing for 3D coordinate calculation through multi-view triangulation. To achieve a full spherical range of camera angles, the object is rotated on a stepper motor, reducing the need for numerous cameras and minimizing background clutter.

A critical aspect of their methodology is the rigorous calibration process. This includes estimating intrinsic camera parameters for each of the 8 cameras using a checkerboard pattern. Extrinsic parameters and relative camera poses are then estimated by capturing checkerboard images from neighboring cameras and solving for poses using the Perspective-n-Point (PnP) algorithm. To mitigate accumulated errors from successive pose estimations, they employ a gradient descent optimization process that minimizes reprojection loss and accumulation loss.

For 3D reconstruction, they process binary-coded images to extract unique key points, which are then used for triangulation with the calibrated camera parameters to generate a 3D point cloud. Outlier filtering is applied to refine the point cloud.

Datasets

The paper introduces the FineView Dataset.

  • Content: The dataset comprises 213 butterfly objects across 173 species, with each specimen dry-preserved and pinned. It includes multi-view 2D images (320 camera poses per object), 3D point clouds, object masks, and corresponding key points.
  • Key Features: It offers accurate extrinsic and intrinsic camera parameters, high-resolution images (1.4k x 1.2k to 4.2k x 3.6k pixels), and 3D point clouds (6k to 93k points). The data is captured from a spherical 360-degree view, providing comprehensive coverage.
  • Comparison: The authors compare FineView to other multi-view 2D datasets, highlighting that FineView is the first to offer fine-grained category classes. They also created two additional butterfly FGVC datasets: a Flickr butterfly dataset (150 species, 12k images) and an iNaturalist butterfly dataset (1k+ species, 800k images) for evaluation purposes.

Key Insights and Ideas

  • The necessity of precise 3D and multi-view data for FGVC: The paper strongly argues that human-like understanding of fine-grained categories, especially for objects like insects, requires more than just 2D images. Inferring 3D shape, understanding pose, and having consistent camera views are crucial for robust classification, particularly in challenging scenarios.
  • Custom scanning system for controlled data acquisition: Developing their own scanning system allowed them to achieve precise camera calibration and capture data from a full spherical range, overcoming limitations of existing methods that might rely on less controlled web-scraped data or have sparse camera angle coverage.
  • Structured light for dense 3D information: The use of structured-light patterns, despite requiring more scanning time, is highlighted as essential for obtaining dense 3D information and mapping key points in feature-poor regions, which is a significant advantage over standard feature extraction methods like COLMAP.
  • Optimization for pose accuracy: The refinement of camera poses using gradient descent optimization demonstrates the importance of minimizing accumulated errors in multi-view calibration pipelines to achieve accurate 3D reconstructions.

Key Results

  • Dataset Size: The FineView dataset contains 68,160 images from 173 subcategories of butterflies.
  • Data Collection Efficiency: The system captures approximately 900 images per sample in about 7 minutes, which is faster than previous methods for similar quality data.
  • FGVC Performance: When training a ResNet-50 model with FineView data mixed with other datasets (Flickr, iNaturalist), they observed improved accuracy across various mixing strategies (+random, +consecutive, +pose) compared to using only the other datasets. For instance, adding FineView data to the iNat17 dataset improved accuracy from 76.51% to 77.36%.
  • NeRF Performance: The FineView dataset, when used to train a vanilla NeRF model, achieved better quantitative results (PSNR, SSIM, LPIPS) compared to the MVImgNet dataset, demonstrating its utility for novel view synthesis.

Limitations and Future Work

  • Limited Fine Structures: The authors acknowledge that their system has difficulty precisely capturing very small and fine structures, such as antennae and legs, due to the resolution limitations of the 3D scan.
  • Current Domain: The current dataset is limited to butterflies.
  • Future Work: The authors plan to expand the platform to capture more fine-grained categories and build a larger-scale 3D and multi-view dataset. They envision FineView being used for diverse applications like autonomous field biology, conservation AI, and augmented reality. They also hope to overcome the challenges in capturing fine structures in future work.

Summary updated

BibTeX
@inproceedings{onda2025fineview,
  author = {Suguru Onda and Ryan Farrell},
  title = {{The FineView Dataset:A 3D Scanned Multi-View Object Dataset of Fine-Grained Category Instances}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure