Benchmark Platform for Ultra-Fine-Grained Visual Categorization Beyond Human Performance

Xiaohan Yu, Yang Zhao, Yongsheng Gao, Xiaohui Yuan, Shengwu Xiong

Introduction

Computer vision has made significant strides in Fine-Grained Visual Categorization (FGVC), successfully distinguishing between subordinate categories like specific bird species or car models. However, these tasks typically rely on visual cues that are perceptible to the human eye. This paper introduces Ultra-Fine-Grained Visual Categorization (UFGVC), a new frontier that pushes classification to the "cultivar" level—where differences are so subtle they often elude even human experts.

Hierarchy of categorization from coarse classes to ultra-fine-grained sub-classes.

The Challenge of Ultra-Fine-Grained Categorization

The primary hurdle in UFGVC is that visual differences between cultivars (e.g., different varieties of soybean) are nearly invisible or impossible for a human to articulate. Traditional datasets rely on human labeling, which becomes a bottleneck at this scale. To overcome this, the authors utilized genetic resource banks to provide objective "ground truth" labels, creating a massive dataset of over 47,000 images across 3,526 categories.

The UFG Dataset

The core contribution of this work is the UFG image dataset, which is significantly larger and more complex than previous leaf datasets. It is divided into several subsets to test different aspects of model performance:

  • SoyAgeing & SoyGene: Large-sample subsets (20+ images per category) to evaluate peak performance.
  • SoyGlobal, SoyLocal, & Cotton80: Small-sample subsets (as few as 6 images per category) to test generalization and few-shot learning capabilities.
Representative leaf samples from the various subsets of the UFG dataset.

Benchmarking State-of-the-Art Models

The researchers established a rigorous evaluation pipeline using 13 state-of-the-art classification methods, including:

  1. General Deep Learning: Architectures like ResNet-50 and self-supervised models like SimCLR.
  2. Specialized FGVC Methods: Algorithms like DCL (Destruction and Construction Learning) and MaskCOV, designed to identify subtle local features.

Images were captured in controlled environments using 600 DPI scanners to ensure that models focused purely on morphological features like leaf venation and texture rather than background noise.

Key Findings

Surpassing Human Performance

One of the most striking results is that deep learning models consistently outperformed human breeding professionals. In a 3-category task on the SoyLocal and Cotton80 subsets, humans achieved approximately 66% accuracy, while specialized models like DCL reached 88%. This suggests an "expert ceiling" where AI can detect statistical patterns in texture that humans cannot perceive.

The Importance of Localization

Using Classification Activation Maps (CAM), the study found that successful models focus on consistent, compact regions of the leaf. This indicates that ultra-fine differences are often concentrated in specific local structures rather than the overall leaf shape.

The Small-Sample Struggle

While models excel with ample data, performance drops significantly on small-sample subsets. On the SoyGlobal subset (1,938 categories with 6 images each), the best model only achieved 54.58% accuracy, highlighting that UFGVC remains a challenging few-shot learning problem.

Conclusion and Future Directions

The UFGVC benchmark reveals that while AI can surpass human experts in identifying subtle cultivars, several challenges remain. Models often overfit on small datasets, and their decision-making processes lack interpretability for human scientists. Future work should focus on improving few-shot learning techniques and moving UFGVC from controlled laboratory settings into "in-the-wild" environments like agricultural fields.

Summary updated

Abstract

Deep learning methods have achieved remarkable success in fine-grained visual categorization. Such successful categorization at sub-ordinate level, e.g., different animal or plant species, however relies heavily on the visual differences that human can observe and the ground-truths are labelled on the basis of such human visual observation. In contrast, few research has been done for visual categorization at the ultra-fine-grained level, i.e., a granularity where even human experts can hardly identify the visual differences or are not yet able to give affirmative labels by inferring observed pattern differences. This paper reports our efforts towards mitigating this research gap. We introduce the ultra-fine-grained (UFG) image dataset, a large collection of 47,114 images from 3,526 categories. All the images in the proposed UFG image dataset are grouped into categories with different confirmed cultivar names. In addition, we perform an extensive evaluation of state-of-the-art fine-grained classification methods on the proposed UFG image dataset as comparative baselines. The proposed UFG image dataset and evaluation protocols is intended to serve as a benchmark platform that can advance research of visual classification from approaching human performance to beyond human ability, via facilitating benchmark data of artificial intelligence (AI) not to be limited by the labels of human intelligence (HI). The dataset is available online at https://githuh.com/XiaohanYu-GU/Ultra-FGVC.
BibTeX
@inproceedings{yu2021benchmark,
  author = {Xiaohan Yu and Yang Zhao and Yongsheng Gao and Xiaohui Yuan and Shengwu Xiong},
  title = {{Benchmark Platform for Ultra-Fine-Grained Visual Categorization Beyond Human Performance}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01012},
}
Figure