HEPHA: A Mixed-Initiative Image Labeling Tool for Specialized Domains

Shiyuan Zhou, Bingxuan Li, Xiyuan Chen, Zhi Tu, Yifeng Wang, Yiwen Xiang, Tianyi Zhang

Abstract

Image labeling is an important task for training computer vision models. In specialized domains, such as healthcare, it is expensive and challenging to recruit specialists for image labeling. We propose HEPHA, a mixed-initiative image labeling tool that elicits human expertise via inductive logic learning to infer and refine labeling rules. Each rule comprises visual predicates that describe the image. HEPHA enables users to iteratively refine the rules by either direct manipulation through a visual programming interface or by labeling more images. To facilitate rule refinement, HEPHA recommends which rule to edit and which predicate to update. For users unfamiliar with visual programming, HEPHA suggests diverse and informative images to users for further labeling. We conducted a within-subjects user study with 16 participants and compared HEPHA with a variant of HEPHA and a deep learning-based approach. We found that HEPHA outperforms the two baselines in both specialized-domain and general-domain image labeling tasks.

Introduction

Training modern computer vision models requires massive amounts of labeled data. While general tasks like identifying common objects can be crowdsourced, specialized domains—such as medical imaging or rare species identification—require expert knowledge. Experts are often too busy or expensive for traditional large-scale labeling. Furthermore, existing automated tools often rely on "black-box" deep learning models, which offer no transparency when errors occur. HEPHA addresses these challenges by making labeling logic transparent, editable, and efficient through a mixed-initiative approach.

The HEPHA Framework

HEPHA facilitates a collaborative loop between human experts and AI. The system architecture is built on three core pillars:

  1. Visual Predicate Extraction: The system utilizes pre-trained models (e.g., Grounding-DINO or SAM) to identify specific objects and attributes within images, converting visual data into semantic predicates (e.g., "contains a bed").
  2. Rule Inference via ILP: Instead of standard neural networks, HEPHA employs Inductive Logic Programming (ILP). By analyzing a small set of labeled examples, the system generates human-readable "if-then" rules using Boolean algebra.
  3. Iterative Refinement: This mixed-initiative cycle allows experts to refine the system's logic directly rather than just providing more raw data.
Overview of the HEPHA workflow: from expert bootstrapping to iterative refinement of logic rules.

Interactive Rule Refinement

The core innovation of HEPHA is its visual programming interface. Users can interact with the inferred rules in several ways:

  • Direct Manipulation: Users can drag-and-drop logic blocks to add, remove, or modify predicates within a rule.
  • Constraint Management: Experts can "lock" rules they know are correct to prevent the AI from overwriting them or "ban" specific logic combinations that lead to systematic errors.
  • Intelligent Recommendations: To reduce cognitive load, the system uses a TF-IDF-inspired ranking to suggest which visual features are most critical for a user to consider when editing a rule.
The Label Correction and Rule Editing interface enables direct manipulation of the underlying classification logic.

Evaluation and Results

The authors evaluated HEPHA across four datasets: specialized domains (Bird species and Glaucoma detection) and general domains (Occupations and Indoor Scenes).

Key findings include:

  • Superior Accuracy: HEPHA achieved 87% to 91% accuracy across all tasks, notably outperforming a ResNet18 baseline by 42% in specialized domains.
  • Data Efficiency: The system reached peak performance using only 16%–19.5% of the dataset, significantly less than required by deep learning approaches.
  • User Experience: 14 out of 16 study participants preferred HEPHA, noting that the rule-editing process was engaging and provided a sense of control and transparency.

Discussion and Future Work

While highly effective, HEPHA currently uses relatively simple Disjunctive Normal Form logic. Future iterations could support more complex, nested logic structures. Additionally, the system's performance is currently bound by the quality of the underlying visual predicate extractors. The authors suggest that this logic-based, mixed-initiative approach could be extended beyond images to other modalities like text and document classification.

Summary updated

Abstract

Image labeling is an important task for training computer vision models. In specialized domains, such as healthcare, it is expensive and challenging to recruit specialists for image labeling. We propose HEPHA, a mixed-initiative image labeling tool that elicits human expertise via inductive logic learning to infer and refine labeling rules. Each rule comprises visual predicates that describe the image. HEPHA enables users to iteratively refine the rules by either direct manipulation through a visual programming interface or by labeling more images. To facilitate rule refinement, HEPHA recommends which rule to edit and which predicate to update. For users unfamiliar with visual programming, HEPHA suggests diverse and informative images to users for further labeling. We conducted a within-subjects user study with 16 participants and compared HEPHA with a variant of HEPHA and a deep learning-based approach. We found that HEPHA outperforms the two baselines in both specialized-domain and general-domain image labeling tasks. Our code is available at https://github.com/Neural-Symbolic-Image-Labeling/NSILWeb.
BibTeX
@inproceedings{zhou2025hepha,
  author = {Shiyuan Zhou and Bingxuan Li and Xiyuan Chen and Zhi Tu and Yifeng Wang and Yiwen Xiang and Tianyi Zhang},
  title = {{HEPHA: A Mixed-Initiative Image Labeling Tool for Specialized Domains}},
  booktitle = {IUI},
  year = {2025},
  doi = {10.1145/3708359.3712072},
}
Figure