Abstract
Image labeling is an important task for training computer vision models. In specialized domains, such as healthcare, it is expensive and challenging to recruit specialists for image labeling. We propose HEPHA, a mixed-initiative image labeling tool that elicits human expertise via inductive logic learning to infer and refine labeling rules. Each rule comprises visual predicates that describe the image. HEPHA enables users to iteratively refine the rules by either direct manipulation through a visual programming interface or by labeling more images. To facilitate rule refinement, HEPHA recommends which rule to edit and which predicate to update. For users unfamiliar with visual programming, HEPHA suggests diverse and informative images to users for further labeling. We conducted a within-subjects user study with 16 participants and compared HEPHA with a variant of HEPHA and a deep learning-based approach. We found that HEPHA outperforms the two baselines in both specialized-domain and general-domain image labeling tasks.
Introduction
Training modern computer vision models requires massive amounts of labeled data. While general tasks like identifying common objects can be crowdsourced, specialized domains—such as medical imaging or rare species identification—require expert knowledge. Experts are often too busy or expensive for traditional large-scale labeling. Furthermore, existing automated tools often rely on "black-box" deep learning models, which offer no transparency when errors occur. HEPHA addresses these challenges by making labeling logic transparent, editable, and efficient through a mixed-initiative approach.
The HEPHA Framework
HEPHA facilitates a collaborative loop between human experts and AI. The system architecture is built on three core pillars:
- Visual Predicate Extraction: The system utilizes pre-trained models (e.g., Grounding-DINO or SAM) to identify specific objects and attributes within images, converting visual data into semantic predicates (e.g., "contains a bed").
- Rule Inference via ILP: Instead of standard neural networks, HEPHA employs Inductive Logic Programming (ILP). By analyzing a small set of labeled examples, the system generates human-readable "if-then" rules using Boolean algebra.
- Iterative Refinement: This mixed-initiative cycle allows experts to refine the system's logic directly rather than just providing more raw data.
Interactive Rule Refinement
The core innovation of HEPHA is its visual programming interface. Users can interact with the inferred rules in several ways:
- Direct Manipulation: Users can drag-and-drop logic blocks to add, remove, or modify predicates within a rule.
- Constraint Management: Experts can "lock" rules they know are correct to prevent the AI from overwriting them or "ban" specific logic combinations that lead to systematic errors.
- Intelligent Recommendations: To reduce cognitive load, the system uses a TF-IDF-inspired ranking to suggest which visual features are most critical for a user to consider when editing a rule.
Evaluation and Results
The authors evaluated HEPHA across four datasets: specialized domains (Bird species and Glaucoma detection) and general domains (Occupations and Indoor Scenes).
Key findings include:
- Superior Accuracy: HEPHA achieved 87% to 91% accuracy across all tasks, notably outperforming a ResNet18 baseline by 42% in specialized domains.
- Data Efficiency: The system reached peak performance using only 16%–19.5% of the dataset, significantly less than required by deep learning approaches.
- User Experience: 14 out of 16 study participants preferred HEPHA, noting that the rule-editing process was engaging and provided a sense of control and transparency.
Discussion and Future Work
While highly effective, HEPHA currently uses relatively simple Disjunctive Normal Form logic. Future iterations could support more complex, nested logic structures. Additionally, the system's performance is currently bound by the quality of the underlying visual predicate extractors. The authors suggest that this logic-based, mixed-initiative approach could be extended beyond images to other modalities like text and document classification.
Summary updated