HEPHA: A Mixed-Initiative Image Labeling Tool for Specialized Domains

Shiyuan Zhou, Bingxuan Li, Xiyuan Chen, Zhi Tu, Yifeng Wang, Yiwen Xiang, Tianyi Zhang

TL;DR

HEPHA is a "mixed-initiative" image labeling tool designed to help experts label complex datasets (like medical scans or rare species) with minimal effort. By using Inductive Logic Programming (ILP) to turn a few labeled examples into human-readable logic rules, it allows experts to "teach" the system through a visual drag-and-drop interface rather than just labeling thousands of individual images.

What Problem Does This Paper Address?

Training modern computer vision models requires massive amounts of labeled data. For general tasks (like identifying cats vs. dogs), we can use crowdsourcing. However, for specialized domains like healthcare or ecology, you need experts—and experts are expensive and busy.

Existing automated labeling tools often rely on "black-box" deep learning models. When these models make a mistake, an expert has no way to see why it happened or how to fix the underlying logic. This creates a "use barrier" where experts struggle to trust or refine the automation. HEPHA aims to bridge this gap by making the labeling logic transparent, editable, and efficient even with very little starting data.

Methods

HEPHA uses a mixed-initiative approach, meaning the human and the AI collaborate in a loop. The system architecture consists of three main parts:

  1. Visual Predicate Extraction: HEPHA uses pre-trained models (like Grounding-DINO or SAM) to identify objects and attributes in images (e.g., "contains a tree root," "has a yellow breast").
  2. Rule Inference (ILP): Instead of standard deep learning, HEPHA uses Inductive Logic Programming (specifically the FOIL algorithm). It looks at a few labeled images and generates "if-then" rules. For example: If an image contains a 'bed' AND 'lamp', label it as 'bedroom'.
  3. Interactive Refinement: This is the core innovation. Users don't just label more images; they can use a Visual Programming Interface to:
  4. Drag-and-drop new logic blocks to refine rules.
  5. Lock rules they know are correct so the AI doesn't overwrite them.
  6. Ban specific logic combinations that lead to errors.
  7. Saliency-based Recommendations: The system uses a TF-IDF-inspired ranking to suggest which visual features (predicates) are most important for the user to consider when editing a rule.

Datasets

The authors evaluated HEPHA using four distinct datasets to test both specialized and general domain performance:

  • Bird (Specialized): 400 images of four easily confused bird species (e.g., Cerulean vs. Magnolia Warbler).
  • Medical (Specialized): 305 images of eyes, categorized as normal or having glaucoma.
  • Occupation (General): 300 images of chefs, farmers, and teachers.
  • Indoor Scene (General): 300 images of conference rooms, kitchens, and bedrooms.

Key Insights and Ideas

  • Rules over Raw Data: In specialized domains, a single expert-defined rule is often worth more than hundreds of labeled images. By allowing experts to edit the logic directly, the system reaches high accuracy much faster than deep learning baselines.
  • Interpretability as a Debugger: Because the rules are written in plain logic (Boolean algebra), experts can diagnose exactly why a "Mangrove Forest" was mislabeled as a "Boreal Forest" and fix the specific predicate (e.g., "Not Has Water Pool").
  • The "Fun" Factor: Participants in the study found the rule-editing feature engaging, describing it as "playing around" with the logic, which reduced the perceived cognitive load of a traditionally tedious task.
  • Active Learning Integration: For users who aren't comfortable with logic, HEPHA uses a multi-criteria active learning algorithm to suggest the most "informative" and "diverse" images for manual labeling, which then helps the AI regenerate better rules automatically.

Key Results

The authors compared HEPHA against a standard ResNet18 model and a version of HEPHA without rule-editing features:

  • Superior Accuracy: HEPHA achieved 87% to 91% accuracy across all tasks. In specialized domains, it outperformed the ResNet baseline by a staggering 42%.
  • Data Efficiency: HEPHA required significantly fewer labeled images (only 16%–19.5% of the dataset) to reach peak performance compared to the deep learning approach.
  • Time Savings: Users spent 3 to 7 minutes less labeling specialized datasets with HEPHA than with the ResNet-based tool.
  • User Preference: 14 out of 16 participants preferred HEPHA over the baselines, citing the control and transparency provided by the rule editor.

Limitations and Future Work

  • Rule Complexity: Currently, HEPHA uses a relatively simple "Disjunctive Normal Form" (ORs of ANDs). Users expressed a desire for more complex, nested logic (e.g., "If A is true, then either B or C must be true, but not both").
  • Predicate Dependency: The system’s success relies heavily on the quality of the pre-trained visual extractors. If the underlying AI can't "see" a specific feature (like a specific type of cell structure), the expert can't write a rule for it.
  • Scalability: While effective for 3-4 classes, the authors acknowledge that managing rules for datasets with dozens or hundreds of categories might become visually overwhelming for users.
  • Future Modalities: The authors suggest extending this logic-based approach to other data types, such as text labeling or document classification.

Summary updated

Abstract

Image labeling is an important task for training computer vision models. In specialized domains, such as healthcare, it is expensive and challenging to recruit specialists for image labeling. We propose HEPHA, a mixed-initiative image labeling tool that elicits human expertise via inductive logic learning to infer and refine labeling rules. Each rule comprises visual predicates that describe the image. HEPHA enables users to iteratively refine the rules by either direct manipulation through a visual programming interface or by labeling more images. To facilitate rule refinement, HEPHA recommends which rule to edit and which predicate to update. For users unfamiliar with visual programming, HEPHA suggests diverse and informative images to users for further labeling. We conducted a within-subjects user study with 16 participants and compared HEPHA with a variant of HEPHA and a deep learning-based approach. We found that HEPHA outperforms the two baselines in both specialized-domain and general-domain image labeling tasks. Our code is available at https://github.com/Neural-Symbolic-Image-Labeling/NSILWeb.
BibTeX
@inproceedings{zhou2025hepha,
  author = {Shiyuan Zhou and Bingxuan Li and Xiyuan Chen and Zhi Tu and Yifeng Wang and Yiwen Xiang and Tianyi Zhang},
  title = {{HEPHA: A Mixed-Initiative Image Labeling Tool for Specialized Domains}},
  booktitle = {IUI},
  year = {2025},
  doi = {10.1145/3708359.3712072},
}

Loading related content...

Figure