TL;DR
This paper surveys how machine learning (ML) researchers elicit domain knowledge from experts, developing a taxonomy to categorize these approaches. They found that elicitation practices often lack transparency and systematicity, highlighting opportunities for more rigorous and reproducible methods to build better ML models.
What Problem Does This Paper Address?
Machine learning models are increasingly used in real-world applications, and incorporating domain expertise is crucial for building robust, accurate, and trustworthy systems. However, the process of eliciting this knowledge from domain experts is often challenging. Experts may have different ways of thinking and communicating compared to ML practitioners, and building common ground is essential. Despite the importance of this process, there's a lack of systematic study on how ML researchers actually elicit knowledge from domain experts. This can lead to ad-hoc methods, potentially resulting in unreliable elicited knowledge and hindering the full potential of ML. This paper aims to fill this gap by characterizing current practices and identifying areas for improvement.
Methods
The authors conducted a systematic literature review of 28 research papers published between 1995 and 2020 that explicitly describe the elicitation of domain knowledge for ML model development. They developed a novel taxonomy to categorize these elicitation approaches, consisting of four main levels:
- Elicitation Goal: The purpose of the elicitation (e.g., problem specification, feature engineering, model development, model evaluation).
- Elicitation Target: The type of knowledge being elicited (e.g., background knowledge, labels, feature explanations, model constraints).
- Elicitation Process: How the knowledge is elicited (e.g., medium, prompts, response format, number of experts, validation).
- Use of Elicited Knowledge: How the obtained knowledge is integrated into the ML pipeline.
They then analyzed 73 "elicitation paths" (unique combinations of codes within the taxonomy) across the selected papers to identify trends and patterns.
Datasets
Not discussed in the paper. The study focuses on analyzing existing research papers and does not introduce new datasets or benchmarks.
Key Insights and Ideas
- Elicitation is a Black Box: A significant portion of the surveyed papers (33%) had an "unknown" elicitation medium, meaning crucial details about how knowledge was gathered were missing. This lack of transparency makes it difficult to reproduce or evaluate the elicitation process.
- Model Development Dominates: The most common elicitation goal was "model development" (38% of paths), suggesting researchers are primarily focused on using expert knowledge to define model structures and parameters.
- Feature Engineering Lags: While feature engineering is important, it was a less frequent elicitation goal (22%) compared to model development. This might indicate a missed opportunity to leverage expert insights earlier in the pipeline.
- Lack of Rigor: Many elicitation processes were found to be "not well-defined" (32% of paths), and responses were often "unvalidated" (75% of paths). This suggests a tendency towards less systematic and scientifically rigorous methods.
- Limited Use of Established Protocols: Despite the existence of well-developed protocols for expert elicitation in other fields (e.g., SHELF, Delphi), only one paper explicitly mentioned using such a protocol.
- Cognitive Bias is Underaddressed: While some papers acknowledged cognitive biases, only a few explicitly attempted to mitigate them, indicating a gap in addressing potential expert biases.
Key Results
The analysis of 73 elicitation paths revealed several trends:
- Problem Specification: Primarily involved eliciting background knowledge and processes (39%) and labels (35%). These paths often lacked well-defined prompts and had unconstrained response formats.
- Feature Engineering: Most frequently targeted feature relevance (38%) and feature explanations. While prompts were often well-defined, the use of elicited knowledge was frequently "not well-defined" (63%), suggesting researchers had discretion in how they used this information.
- Model Development: Heavily focused on feature relationships (39%) and feature relevance. A significant portion of these paths (54%) accounted for unreliable responses, and the use of elicited knowledge was generally well-defined. However, the elicitation medium was often unknown (54%).
- Model Evaluation: The least common goal, with a mix of well-defined and not-well-defined uses of elicited knowledge.
Limitations and Future Work
The authors acknowledge several limitations:
- Sample Size and Scope: The study analyzed 28 papers, which, while comprehensive, may not represent the entirety of ML research involving domain expertise. They encourage more observational research.
- Implicit Elicitation: The study focused on explicit elicitation, excluding cases where domain experts are part of the research team and their knowledge is implicitly integrated. Future work could explore these inter-disciplinary team dynamics.
- Deep Learning Underrepresentation: The sample included only a few papers using neural networks, suggesting a need for more research on elicitation for deep learning models.
The authors propose several avenues for future work:
- Systematizing and Evaluating Elicitation: Developing and evaluating more systematic and rigorous elicitation methods.
- Improving Transparency and Traceability: Encouraging detailed documentation of elicitation processes and the use of elicited knowledge.
- Establishing Context and Common Ground: Investigating how to better introduce ML goals and perspectives to domain experts.
- Addressing Cognitive Bias: Drawing on broader elicitation literature to better anticipate and mitigate expert biases.
- Validating Elicited Information: Exploring more robust methods for validating expert knowledge, such as through aggregation of multiple experts' responses.
- Typologizing Common Formats: Better distinguishing the use of common knowledge formats (like instances or feature relationships) across different ML pipeline stages.
Summary updated