Expert-sourcing Domain-specific Knowledge: The Case of Synonym Validation

Michael Unterkalmsteiner, Andrew YatesREFSQ Workshops 2019 · OpenReview

Introduction

Supervised machine learning models rely heavily on the quality of their training data. While general-purpose crowdsourcing platforms like Amazon Mechanical Turk are effective for simple tasks like image labeling, they often fail when tasks require deep domain expertise. In specialized fields such as software engineering or the construction industry, the "crowd" must be replaced by experts. This paper presents an "expert-sourcing" framework designed to validate over 1.4 million synonym candidates for the Swedish construction industry's classification system, CoClass.

The Challenge of Domain Expertise

The CoClass system is a hierarchical ontology used to standardize terminology across the construction business. To enhance automated requirements engineering, the system requires a comprehensive list of synonyms for its 1,430 core terms. However, initial attempts using automated "learning-to-rank" models yielded a precision of only 1%. This resulted in a massive dataset of 1.43 million potential synonym pairs that required human validation.

Because the terminology is highly technical, general workers cannot distinguish between valid synonyms and superficially similar terms. The task necessitates domain specialists who are often busy professionals with little time for repetitive data labeling. The research focuses on how to design a tool that maintains engagement and ensures accuracy among these experts.

The Expert-Sourcing Framework

The researchers extended Pybossa, an open-source crowdsourcing framework, to create a specialized validation tool. The interface is designed to minimize cognitive load while maximizing the utility of the expert's time. The workflow is divided into two primary phases:

  1. Selection Phase: The expert is presented with a target term, its full hierarchical context within the ontology (e.g., Components > Limiting objects > Fence), and its official definition. They then select valid synonyms from a list of candidates.
  2. Results/Feedback Phase: Immediately following submission, the expert receives feedback showing how their selections align with existing data and the opinions of their peers.
The expert-sourcing interface during the Selection Phase, providing hierarchical context and definitions for target terms.

To further streamline the process, the tool employs Affinity Propagation Clustering based on Levenshtein distance. By grouping similar-looking candidate synonyms, the tool allows experts to process batches of related terms, preventing the fatigue associated with seeing the same root word repeatedly in a randomized list.

Motivational Design

Traditional crowdsourcing relies on micropayments, which are rarely effective for high-value experts. Instead, the authors mapped their UI features to a psychological model of worker motivation to foster intrinsic and extrinsic engagement.

Taxonomy of worker motivation in crowdsourcing, used to map specific UI features to psychological drivers.

Key motivational strategies include:

  • Human Capital Advancement: By displaying the term's hierarchy and definition, the tool serves as a training mechanism, helping experts better understand the CoClass system they use in their daily work.
  • Direct Job Feedback: The results phase provides immediate social validation by showing how the expert's choices compare to those of their colleagues.
  • Counteracting Bias: Given the low (1%) precision of the automated candidates, experts might become discouraged by long strings of irrelevant terms. The researchers "seed" known true synonyms into the lists to ensure experts find valid matches regularly, maintaining a sense of progress.

Preliminary Results and Future Directions

Early findings confirm that human validation is essential, as the automated baseline remains insufficient for production-level ontologies. The authors have successfully integrated their motivational design into the Pybossa framework and are scaling the study to approximately 500 participants.

Future work will explore gamification elements, such as leaderboards, to further increase long-term engagement. While currently focused on Swedish construction synonyms, the methodology is designed to be generalizable to other expert-heavy tasks, such as identifying code smells or evaluating the ambiguity of software requirements.

Summary updated

BibTeX
@inproceedings{unterkalmsteiner2019expert,
  author = {Michael Unterkalmsteiner and Andrew Yates},
  title = {{Expert-sourcing Domain-specific Knowledge: The Case of Synonym Validation}},
  booktitle = {REFSQ Workshops},
  year = {2019},
  url = {https://openreview.net/forum?id=EX9J1bopES},
}
Figure