A Laypeople Study on Terminology Identification across Domains and Task Definitions

Anna Hätty, Sabine Schulte im WaldeNAACL 2018 · DOI

Introduction

Terminology identification—the process of extracting words or phrases that characterize a specific domain—is a foundational task in Natural Language Processing (NLP). Traditionally, this has been viewed as the exclusive domain of experts, such as terminologists or subject matter specialists. However, professional background often introduces bias, leading to significant disagreement even among experts. This paper investigates whether a "common, natural understanding" of terminology exists among laypeople and how this understanding is shaped by task framing and linguistic properties.

Methodology and Data

The researchers conducted a comprehensive study involving 22 laypeople annotating German texts. To capture a broad spectrum of behavior, the study utilized four distinct task definitions:

  • Domain-Specific Phrases (DS): Highlighting any phrase characteristic of the domain.
  • Indexing (IND): Selecting terms suitable for a back-of-the-book index.
  • Translation Lexicon (TR): Identifying unknown or difficult words for a translation task.
  • Glossary (GL): Selecting terms that require a formal definition.

These tasks were applied across four diverse domains: DIY (do-it-yourself), cooking, hunting, and chess. The dataset consists of 20 German open-source texts (approximately 3,000 words) sourced from wikiHow, Wikibooks, and Wikipedia.

The Impact of Task Framing

One of the study's primary findings is the "Task Effect." The specific framing of the annotation task significantly influences the volume and nature of the identified terms. The DS task, being the most intuitive, yielded the broadest set of terms. In contrast, the GL task resulted in the most conservative selection, as annotators were more selective when considering what warrants a formal definition.

Linguistic Patterns and the Noun Bias

The study revealed a strong preference for nouns among laypeople. Nouns are perceived as the primary carriers of domain-specific information. However, the inclusion of verbs and adjectives increases significantly when they appear within multi-word terms (MWTs). For instance, in the cooking domain, a verb like "beat" might be annotated when part of the phrase "beat the egg yolk."

Figure 6: Distribution of term types (nouns, verbs, adjectives) across different annotation thresholds.

As shown in Figure 6, nouns consistently outnumber other parts of speech across all levels of annotator agreement. While verbs and adjectives are identified, they are far less frequent and their numbers drop off quickly as the requirement for minimum annotations increases.

Inter-Annotator Agreement

The researchers measured consistency using Fleiss’ kappa (κ). Agreement was not uniform across tasks; it was highest for the DS task and lowest for the GL task. This suggests that while people share a core intuition for what "sounds" domain-specific, they struggle more with the subjective decision of what belongs in a formal glossary.

Figure 5: Heatmap showing Fleiss' kappa agreement scores across domains and task definitions.

Figure 5 illustrates this variation. The DS column shows consistently high agreement (indicated by the yellow/lighter colors) across all four domains. This indicates that identifying domain-specific language is a relatively stable and intuitive cognitive task for non-experts, regardless of the specific subject matter.

Challenges: Complexity and Ambiguity

Despite the shared intuition, two main factors impeded agreement:

  1. Structural Complexity: While laypeople are highly consistent in identifying "closed compounds" (common in German, where multiple stems form a single word), they often disagree on the boundaries of multi-word terms (phrases).
  2. Semantic Ambiguity: Words that have both a general-language meaning and a technical meaning (e.g., "solution") are harder for laypeople to categorize. The common usage often "masks" the technical significance for non-experts.

Conclusion and Future Directions

The study demonstrates that laypeople possess a robust, shared understanding of terminology, particularly when tasks are framed around general domain-specificity. While experts remain necessary for highly technical "deep-end" terms, laypeople can provide high-quality annotations for broader applications. Future work in automated term extraction could benefit from modeling how humans prioritize individual subterms within complex phrases, potentially bridging the gap between human intuition and machine performance.

Summary updated

BibTeX
@inproceedings{hatty2018laypeople,
  author = {Anna Hätty and Sabine Schulte im Walde},
  title = {{A Laypeople Study on Terminology Identification across Domains and Task Definitions}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
  doi = {10.18653/v1/n18-2052},
}
Figure