Incremental Image Labeling Via Iterative Refinement

Fausto Giunchiglia, Xiaolei Diao, Mayukh Bagchi

Abstract

Current image recognition datasets often suffer from the "Semantic Gap Problem," where the mapping between visual data and linguistic labels is inconsistent or ambiguous. This paper presents a novel methodology grounded in Knowledge Representation (KR) to refine image labeling through an iterative process. By focusing on visual properties—specifically genus and differentia—rather than simple lexical terms, the proposed approach significantly enhances inter-annotator agreement and improves the classification accuracy of standard deep learning architectures.

Introduction

The efficacy of computer vision models is fundamentally constrained by the quality of their training data. Despite the prevalence of large-scale datasets like ImageNet, systematic flaws in labeling persist, leading to a performance ceiling. These flaws often stem from the "Semantic Gap Problem" (SGP), which describes the disconnect between raw visual information and the high-level linguistic labels assigned by human annotators. This paper argues that moving from subjective, word-based labeling to a rigorous, property-based classification system can mitigate these issues and provide a more robust foundation for machine learning.

The Semantic Gap Problem

The SGP manifests in two primary forms: object ambiguity and visual ambiguity.

Object ambiguity occurs when a scene contains multiple distinct entities, but the dataset provides only a single, often arbitrary, label. For instance, an image depicting an "oboe" within an "orchestra" might be labeled as either, leading to inconsistency across the dataset.

Visual ambiguity arises from the subjective nature of language; labels like "sports car" and "convertible" may overlap, or a term like "ice" might refer to disparate visual contexts such as an iceberg or an ice cube. Traditional labeling relies on "lexical labels" derived from taxonomies like WordNet. However, these labels are frequently influenced by the annotator's cultural background and personal interpretation, introducing noise into the "gold standard" ground truth.

Methodology: Iterative KR-Based Refinement

To address these challenges, we propose a two-step labeling strategy rooted in Knowledge Representation (KR) theory and teleosemantics. This method distinguishes between the "substance" (the physical object) and the "concept" (the linguistic label).

Step 1: Object Localization (S1)

The first step involves identifying and cropping the "main object" within a cluttered scene. By isolating the primary subject, we create "good images" that minimize object ambiguity and ensure the model focuses on the intended substance rather than background noise.

Step 2: Visual Classification (S2)

Rather than selecting a label from a flat list, annotators follow a structured hierarchy based on two KR principles:

  1. Visual Genus: Identifying properties shared across a broad category (e.g., "a marine creature with a carapace").
  2. Visual Differentia: Identifying specific properties that distinguish a sub-category from its peers (e.g., "the presence of claws" to differentiate an American lobster from a Spiny lobster).

Iterative Refinement Loops

The process is managed through three distinct loops to ensure precision:

  • Top-level Loop: Filters the input to ensure only "good" (single-object) images proceed to classification.
  • Vertical Loop: Guides the annotator through a hierarchical refinement of the label, layer by layer, from general to specific.
  • Horizontal Loop: Requires the annotator to compare the object against sibling categories at the same hierarchical level to ensure the most accurate fit.

Experimental Evaluation

The methodology was evaluated using a subset of 191 images from ImageNet, specifically focusing on nine categories of musical instruments. This "Refine" dataset was compared against the original ImageNet labels to measure improvements in both human consistency and machine learning performance.

Inter-annotator Agreement

To measure the reliability of the KR-based approach, two experts relabeled the dataset. The process yielded a Krippendorff’s alpha of 0.9832, indicating near-perfect agreement and a significant reduction in the subjectivity typically associated with image labeling.

Model Performance

We trained five standard computer vision models on both the original ImageNet subset and the Refine dataset. The results demonstrated a consistent performance boost across all architectures when using the refined labels:

  • GoogleNet: Accuracy increased from 72.7% to 82.5%.
  • ResNet: Accuracy increased from 53.8% to 74.1%.
  • VGG: Accuracy increased from 69.9% to 76.2%.

These improvements suggest that the errors in current datasets are not merely random noise but structural deficiencies that can be rectified through systematic labeling logic.

Conclusion and Future Work

This research demonstrates that a property-based, iterative approach to image labeling can bridge the semantic gap and significantly enhance model performance. While the current study was limited to a small subset of musical instruments, the results provide a compelling proof of concept. Future work will focus on scaling this labor-intensive process to larger, more diverse datasets to establish new benchmarks for the computer vision community that prioritize visual properties over ambiguous lexical terms.

Summary updated

BibTeX
@inproceedings{giunchiglia2023incremental,
  author = {Fausto Giunchiglia and Xiaolei Diao and Mayukh Bagchi},
  title = {{Incremental Image Labeling Via Iterative Refinement}},
  booktitle = {Workshops @ ICASSP},
  year = {2023},
  url = {https://openreview.net/forum?id=LBz5sxkpq95},
}
Figure