Expert-sourcing Domain-specific Knowledge: The Case of Synonym Validation

Michael Unterkalmsteiner, Andrew YatesarXiv 2023 · DOI · arXiv

Introduction

Training machine learning models for niche industries presents a significant hurdle: generic crowdsourcing platforms often lack the domain-specific expertise required to handle specialized jargon. While supervised learning is highly dependent on the quality of training data, platforms like Amazon Mechanical Turk are better suited for general tasks, such as image recognition, rather than technical validation. In the construction industry, for instance, identifying whether a "barrier" is synonymous with a specific type of "fence" requires deep professional knowledge.

This paper explores an "expert-sourcing" framework designed to validate synonyms for CoClass, a massive hierarchical classification system used in the construction sector. Currently, CoClass is incomplete, and while automated "learning-to-rank" algorithms can suggest potential synonyms, their precision is remarkably low—approximately 1%. This leaves researchers with the monumental task of manually reviewing 1.43 million synonym candidates. The authors propose a system that leverages specialized task design and motivational psychology to encourage busy professionals to volunteer their expertise for data cleaning.

Methodology and System Design

The researchers adapted Pybossa, an open-source crowdsourcing framework, to create a specialized validation tool. The system architecture is built around three core pillars: task design, computational efficiency, and psychological engagement.

Two-Phase Task Design

The validation process is split into two distinct phases to maximize clarity and provide immediate value to the expert:

  1. Phase 1 (Selection): Experts are presented with a target term, its definition, and its specific location within the CoClass hierarchy. This hierarchical "path" (e.g., Components >> Limiting objects >> Fence) provides the vital context needed to make accurate professional judgments.
  2. Phase 2 (Feedback): Immediately after submission, the expert receives feedback showing how their choices align with those of their peers and the existing database.
The selection interface provides experts with the target term, its definition, and its hierarchical context within CoClass.

Computational Efficiency via Clustering

To combat "labeling fatigue," the tool employs Affinity Propagation Clustering. By using Levenshtein distance—a metric measuring the character changes required to transform one word into another—the system groups similar candidate synonyms. This allows experts to accept or reject entire clusters of terms simultaneously, significantly reducing the cognitive load of sifting through millions of candidates.

Motivational Modeling

A central contribution of this research is the mapping of tool features to a formal model of worker motivation. The authors argue that experts are not merely data validators; they are active learners. By integrating specific design elements, the tool addresses both intrinsic and extrinsic motivational drivers.

The framework maps specific system features to a hierarchy of intrinsic and extrinsic motivations to ensure long-term expert engagement.

For example, displaying the hierarchical path of a term fosters "Human Capital Advancement" by helping experts build a more structured professional vocabulary. Meanwhile, showing alignment with peers serves as "Direct Job Feedback," turning a repetitive labeling task into a form of professional benchmarking. This "Aha!" moment of social alignment acts as a powerful hook for continued participation.

Dataset and Implementation

The study focuses on the CoClass ontology, utilizing:

  • 1,430 target terms from the construction domain.
  • 1.43 million synonym candidates (1,000 per term) generated by a preliminary NLP model.
  • A focus on the Swedish language, which necessitates an expert-sourcing approach due to the limited pool of qualified speakers compared to English-centric datasets.

To maintain quality control, the system utilizes "seeding." By occasionally inserting known true synonyms into the candidate list, the system can monitor expert attention; if an expert consistently misses these seeds, it signals a potential lack of engagement or accuracy.

Key Results and Insights

The paper highlights the stark contrast between automated and human-led validation. While the automated baseline for synonym detection in this domain has a precision of only ~1%, the expert-sourcing framework provides a scalable path toward high-quality data. The framework is designed to scale from a small pilot to a deployment involving approximately 500 participants. Furthermore, by tracking the time experts spend on each task, the authors provide a method for management to quantify the exact cost and investment required to build high-fidelity, domain-specific datasets.

Limitations and Future Work

While successful in synonym validation, the authors acknowledge several areas for expansion. The current task is inherently rigid, offering little room for the "task autonomy" or creativity that often drives intrinsic motivation. Future iterations may include "hard" gamification features, such as leaderboards or level systems, to enhance long-term engagement. Additionally, the researchers suggest that this expert-sourcing logic could be applied to other complex technical tasks, such as identifying code smells in software engineering or evaluating ambiguity in requirements documentation.

Summary updated

Abstract

One prerequisite for supervised machine learning is high quality labelled data. Acquiring such data is, particularly if expert knowledge is required, costly or even impossible if the task needs to be performed by a single expert. In this paper, we illustrate tool support that we adopted and extended to source domain-specific knowledge from experts. We provide insight in design decisions that aim at motivating experts to dedicate their time at performing the labelling task. We are currently using the approach to identify true synonyms from a list of candidate synonyms. The identification of synonyms is important in scenarios were stakeholders from different companies and background need to collaborate, for example when defining and negotiating requirements. We foresee that the approach of expert-sourcing is applicable to any data labelling task in software engineering. The discussed design decisions and implementation are an initial draft that can be extended, refined and validated with further application.
BibTeX
@misc{unterkalmsteiner2023expert,
  author = {Michael Unterkalmsteiner and Andrew Yates},
  title = {{Expert-sourcing Domain-specific Knowledge: The Case of Synonym Validation}},
  year = {2023},
  eprint = {2309.16798},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.16798},
}
Figure