A Survey of Human-in-the-loop for Machine Learning

Xingjiao Wu, Luwei Xiao, Yixuan Sun, Junhang Zhang, Tianlong Ma, Liang He

TL;DR

This survey explores the growing field of Human-in-the-loop (HITL) machine learning, where human intelligence is strategically integrated into the AI development cycle to overcome the "data bottleneck." By combining human domain expertise with machine efficiency, HITL approaches aim to build more accurate, robust, and explainable models at a significantly lower cost than traditional supervised learning.

What Problem Does This Paper Address?

Deep learning has achieved remarkable success, but it has a massive "data hunger." As models grow to include hundreds of millions (or even billions) of parameters, the demand for high-quality labeled data has outpaced our ability to produce it. Furthermore, deep learning models often act as "black boxes," lacking the common-sense reasoning and domain expertise that humans possess.

The authors address the challenge of how to effectively bridge this gap. They argue that instead of trying to automate everything, we should design systems where humans and machines collaborate. This is particularly crucial in high-stakes fields like medicine or security, where data is sparse and the cost of errors is high.

Methods

The paper categorizes HITL research into three progressive levels based on where the human interacts with the pipeline:

  1. Data Processing: This is the most common form of HITL. It includes Data Preprocessing (using humans to guide AutoML tools), Data Annotation (using active learning to identify which samples a human should label first), and Iterative Labeling (a cyclic process where the model's own predictions help humans label data faster).
  2. Model Training and Inference: Here, humans provide more than just labels. They might provide "rationales" (explaining why a classification is correct) or act as a reward function in a Reinforcement Learning (RL) framework. As shown in Figure 5, the human annotator provides rewards ($R_t$) and state feedback ($S_t$) to guide the machine agent's actions.
  3. System Construction: The authors review how HITL is applied in end-to-end systems for specific tasks like security (fact-checking), code production (automated bug repair), and interactive search engines.
Fig. 5: The human-in-the-loop framework based on reinforce- ment learning.

Datasets

As a survey paper, it does not introduce a single new dataset but reviews results across many standard benchmarks, including:

  • NLP: Reddit TL;DR and CNN/DM (summarization), CCG parsing datasets, and various Q&A corpora.
  • Computer Vision: Pedestrian Re-Identification (Re-ID) datasets, 3D point cloud object detection, and medical imaging (e.g., Electron Microscopy).
  • Specialized: Datasets for network anomaly detection and "manga" generation from illustrations.

Key Insights and Ideas

  • The Efficiency Loop: HITL isn't just about getting more labels; it's about getting the right labels. By using active learning to pick "key samples" that the model is most uncertain about, researchers can achieve higher accuracy with 60-70% less manual effort.
  • Explainability through Interaction: In NLP, asking humans to provide rationales for their labels helps researchers peek inside the "black box." This feedback loop (illustrated in Figure 6) makes models more interpretable and robust against adversarial attacks.
  • The Multi-Modal Path for CV: While HITL is well-established in NLP (where humans can easily provide text-based feedback), it is harder to implement in Computer Vision. The authors suggest that multi-modal approaches—using text to describe visual features—are the future of HITL for CV.
  • RL as a Natural Fit: Reinforcement Learning provides a formal framework for HITL, where the human effectively becomes part of the environment, providing the "reward" that shapes the model's behavior.
Fig. 6: The model training and inferencing workflow of human-in-the-loop in Natural Language Processing.

Key Results

The survey highlights impressive quantitative gains across various domains:

  • NLP: A hybrid human-AI framework (MARTA) for text classification improved accuracy from 0.84 to 0.96. Interactive semantic parsing saw gains from 0.61 to 0.72.
  • Computer Vision: In pedestrian detection, HITL approaches boosted recall by 11% while decreasing the number of required manual annotations by 67%.
  • Software Engineering: The LEARN2FIX system for automated program repair achieved 75% accuracy with minimal human queries.

Limitations and Future Work

Despite the progress shown in Figure 2, several hurdles remain:

  • Lack of Benchmarks: There is currently no unified standard for evaluating HITL systems, making it difficult to compare different approaches fairly.
  • Human Factors: The "human" in the loop is not a perfect machine. Issues like user fatigue, varying levels of expertise, and the potential for malicious users to bias models (AI Safety) need more research.
  • Generalization: Most HITL systems are currently designed for narrow, specific tasks. Developing a general multitasking framework that can incorporate human knowledge across different domains remains a major challenge.
Fig. 2: The increasing research interest in the human-in- the-loop, obtained through Google scholar search with key- words: “human-in-the-loop” and “machine learning”.

Summary updated

Abstract

Human-in-the-loop aims to train an accurate prediction model with minimum cost by integrating human knowledge and experience. Humans can provide training data for machine learning applications and directly accomplish tasks that are hard for computers in the pipeline with the help of machine-based approaches. In this paper, we survey existing works on human-in-the-loop from a data perspective and classify them into three categories with a progressive relationship: (1) the work of improving model performance from data processing, (2) the work of improving model performance through interventional model training, and (3) the design of the system independent human-in-the-loop. Using the above categorization, we summarize major approaches in the field; along with their technical strengths/ weaknesses, we have simple classification and discussion in natural language processing, computer vision, and others. Besides, we provide some open challenges and opportunities. This survey intends to provide a high-level summarization for human-in-the-loop and motivates interested readers to consider approaches for designing effective human-in-the-loop solutions.
BibTeX
@article{wu2022survey,
  author = {Xingjiao Wu and Luwei Xiao and Yixuan Sun and Junhang Zhang and Tianlong Ma and Liang He},
  title = {{A Survey of Human-in-the-loop for Machine Learning}},
  journal = {Future Generation Computer Systems},
  year = {2022},
  doi = {10.1016/j.future.2022.05.014},
}

Loading related content...

Figure