Bridging Multi-disciplinary Collaboration Challenges in ML Development via Domain Knowledge Elicitation

Soya ParkNAACL 2021

TL;DR

Building effective ML models in specialized domains is hindered by the difficulty of transferring domain expertise to data scientists. This paper introduces Ziva, an interface designed to bridge this gap by facilitating the structured elicitation of domain concepts and justifications for data labels, making this knowledge more accessible for ML development.

What Problem Does This Paper Address?

Developing machine learning (ML) models for specialized domains like healthcare or finance is a complex and time-consuming endeavor. A major bottleneck is the steep learning curve for data scientists to acquire the necessary domain-specific knowledge. This knowledge is crucial for various stages of ML development, including defining problems, labeling data, engineering features, and debugging models. Currently, the transfer of this knowledge from domain experts to data scientists is often ad-hoc, relying on costly and inefficient iterative conversations. This process rarely yields knowledge in a format readily usable by ML pipelines, leading to frustration and suboptimal model development.

Methods

The paper introduces Ziva, an interface designed to facilitate knowledge sharing between domain experts and data scientists. Ziva focuses on two primary mechanisms:

  • Concept Creation: Domain experts are presented with representative data instances and asked to categorize them using a card-sorting activity. This helps them group general domain concepts (e.g., "food," "service") and then define specific attributes or descriptions related to these concepts (e.g., "tasty," "kind"). This process aims to build a structured taxonomy of domain knowledge.
  • Justification Elicitation: Once concepts are established, domain experts provide justifications for the labels assigned to data instances. Ziva offers five different methods for this elicitation, moving beyond simple text highlighting:
  • Bag of Words: Highlighting text snippets that justify a label.
  • Instance Perturbation: Editing an instance to make its assigned label no longer justifiable.
  • Instance Simplification: Shortening an instance to retain only the text that justifies the label.
  • Concept Bag of Words: Highlighting text snippets and grouping them under previously defined concepts.
  • Concept Annotation: Distinguishing between words relevant to a topic and words relevant to its description within the instance, linking them to concepts.

Ziva aims to provide a structured and asynchronous way for knowledge to be shared and reused across different modeling tasks within a domain.

Datasets

Not discussed in the paper. The paper focuses on the interface design and methodology rather than specific dataset evaluations.

Key Insights and Ideas

  • The "Ad-Hoc" Problem is Real: The paper highlights that current methods for domain knowledge transfer are informal and inefficient, leading to significant challenges in ML development.
  • Structured Elicitation is Key: Moving beyond simple labeling, Ziva's structured approach to concept creation and justification elicitation is crucial for making domain knowledge actionable for data scientists.
  • Multiple Justification Methods Offer Nuance: The variety of justification elicitation methods (simplification, perturbation, concept-based) suggests that different approaches can capture different facets of domain expertise, providing richer information than a single method.
  • Concept Hierarchies are Powerful: The emphasis on creating taxonomies and linking text to these concepts aligns with the understanding that structured knowledge representations are beneficial for various NLP tasks.

Key Results

The paper does not present quantitative results from a formal evaluation of Ziva's effectiveness in building ML models. Instead, it focuses on the design and rationale behind the Ziva interface and its elicitation methods. The authors mention that the justification elicitation interfaces were designed through an iterative process of paper prototyping, examining consistency of answers and usefulness to data scientists.

Limitations and Future Work

  • Evaluation: The paper acknowledges that details of the interface design and evaluation can be found in a separate publication (Park et al., 2021), implying that a comprehensive evaluation of Ziva's impact on ML model performance is not detailed here.
  • Scalability and Usability: While Ziva aims to facilitate asynchronous exchange, the practical scalability and long-term usability for both domain experts and data scientists would require further investigation.
  • Integration with ML Pipelines: The paper states Ziva's output is intended for consumption by data scientists, but the specific mechanisms for integrating this elicited knowledge into actual ML model development workflows are not elaborated upon.

Future work would likely involve empirical studies to quantify the benefits of Ziva in terms of model performance, development time, and user satisfaction. Further development could also explore more advanced methods for knowledge representation and integration.

Summary updated

BibTeX
@inproceedings{park2021bridging,
  author = {Soya Park},
  title = {{Bridging Multi-disciplinary Collaboration Challenges in ML Development via Domain Knowledge Elicitation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure