Abstract
This paper presents Objectness SIMilarity (OSIM), a novel evaluation metric for 3D scenes that explicitly focuses on "objects," which are fundamental units of human visual perception. Existing metrics assess overall image quality, leading to discrepancies with human perception. Inspired by neuropsychological insights, we hypothesize that human recognition of 3D scenes fundamentally involves attention to individual objects. OSIM enables object-centric evaluations by leveraging an object detection model and its feature representations to quantify the "objectness" of each object in the scene. Our user study demonstrates that OSIM aligns more closely with human perception compared to existing metrics. We also analyze the characteristics of OSIM using various approaches. Moreover, we re-evaluate recent 3D reconstruction and generation models under a standardized experimental setup to clarify advancements in this field.
Introduction
Evaluating the quality of 3D reconstructions—such as those produced by Neural Radiance Fields (NeRFs) or 3D Gaussian Splatting—has traditionally relied on 2D image metrics like PSNR, SSIM, and LPIPS. While these metrics are computationally efficient, they often fail to capture the nuances of human visual perception. Humans do not perceive scenes as a collection of independent pixels; rather, we organize visual information into discrete objects. A new study from Hokkaido University introduces Objectness SIMilarity (OSIM), a metric designed to bridge this gap by explicitly focusing on object-level fidelity.
The Problem with Traditional Metrics
Traditional metrics typically average errors across an entire frame. This leads to a significant discrepancy with human judgment: if a model renders a sharp background but a blurry foreground object, humans will rate the quality poorly. However, because the background often occupies the majority of the pixels, traditional metrics like PSNR might award a higher score to a scene with a sharp background and blurry objects than to one with sharp objects and a blurry background.
As shown in the figure above, humans prefer image (a) where the table is sharp, even if the background is blurred. Yet, PSNR and SSIM rate image (b) higher because the background pixels are more "accurate." OSIM addresses this by prioritizing the fundamental units of human perception: the objects themselves.
Methodology: The OSIM Pipeline
The OSIM metric is calculated through a four-step pipeline that integrates object detection and saliency estimation:
- 3D Rendering: Novel-view images are rendered from the 3D model and compared against ground-truth reference images.
- Feature Extraction: A pre-trained object detection model (YOLOX) identifies objects in the scene and extracts intermediate feature maps, specifically from the
dark5layer, which contains rich semantic information. - Local Index Calculation: For each detected object, OSIM calculates the average cosine similarity between the feature maps of the rendered and reference images within the object's bounding box.
- Saliency Weighting: Recognizing that not all objects are equally important to a viewer, OSIM employs a saliency estimation model (GBVS). This weights the scores of individual objects so that salient objects—those most likely to attract human attention—have a greater impact on the final score.
Experimental Validation
The researchers conducted a comprehensive re-evaluation of state-of-the-art 3D models using diverse datasets:
- 3D Reconstruction: Evaluated on Mip-NeRF360, Blender, and Tanks and Temples (TnT).
- 3D Generation: Focused on 30 detectable object classes from the Google Scanned Objects (GSO) dataset.
- User Study: A study with 23 participants established Mean Opinion Scores (MOS) for 84 reconstructed scenes and 180 generated objects.
Key Results
- Superior Human Alignment: In reconstruction tasks, OSIM achieved a Spearman’s rank correlation ($\rho$) of 0.820 with human MOS, significantly outperforming LPIPS (0.702) and CLIP-similarity (0.756).
- Sensitivity to Degradation: OSIM exhibits a consistent, sharp drop in scores as salient objects are blurred, whereas metrics like CLIP-similarity can be inconsistent, sometimes showing improved scores as visual quality decreases.
- Standardized Benchmarking: The study provides a new leaderboard for 13 reconstruction models (including gsplat) and 6 generation models (including LGM and CRM), offering a more perceptually-grounded comparison of current technologies.
Discussion and Insights
OSIM's object-centric approach offers several advantages. By focusing on objects, it captures semantic and structural fidelity that pixel-level (PSNR) or patch-level (LPIPS) metrics often miss. Furthermore, OSIM serves as a powerful diagnostic tool. Because it calculates scores per object, researchers can visualize exactly which elements of a complex scene are failing to reconstruct properly. For example, a "stop sign" might appear acceptable to PSNR but be flagged as low quality by OSIM if its semantic features are unrecognizable to the object detector.
Limitations and Future Work
While OSIM represents a significant step forward, it is currently dependent on the underlying object detection model. Objects belonging to classes not included in the detector's training set may not be evaluated accurately. Additionally, handling heavily overlapping objects remains a challenge. Future research may extend OSIM to 4D dynamic scenes or integrate it directly into the training loop of 3D models to guide iterative refinement of low-quality regions.
The code for OSIM is available at https://github.com/Objectness-Similarity/OSIM.
Summary updated