Semantic Annotation for the Digital Humanities

Anette Frank, Thomas Bögel, Oliver Hellwig, Nils ReiterLinguistic 2012 · DOI

TL;DR

This paper tackles the challenge of ensuring high-quality semantic annotations for Digital Humanities research, which often requires more precision than current NLP systems can provide. They introduce a novel approach using Markov Logic Networks to detect inconsistencies between different levels of semantic annotation (word-level and discourse-level), aiming to flag potential errors for manual review.

What Problem Does This Paper Address?

Digital Humanities research increasingly relies on computational linguistic analysis to uncover patterns in large text corpora. However, the accuracy of automatic semantic annotations (like word sense disambiguation and coreference resolution) often falls short of the high-quality standards needed for reliable empirical findings. Manual annotation is time-consuming and expensive, making it infeasible for large datasets. This paper addresses the gap between the need for precise semantic annotations and the limitations of current automatic methods by proposing a system for consistency control to identify and flag potential errors in these annotations.

Methods

The core of the paper's contribution lies in its novel approach to inter-level consistency control using Markov Logic Networks (MLNs). Instead of relying solely on evidence within a single annotation level (intra-level consistency), they leverage dependencies between different semantic annotation layers. Specifically, they focus on the interaction between:

  • Word Sense Disambiguation (WSD): Assigning the correct meaning to a word in context.
  • Coreference Resolution (CR): Identifying mentions that refer to the same real-world entity.

The key idea is that mentions belonging to the same discourse entity (identified by CR) should have closely related word senses (as determined by WSD). They formalize this by:

  1. Defining "Centroid" Concepts: For each discourse entity, they compute a "centroid" semantic representation based on the word senses of its constituent mentions, using a graph-theoretical "key player" approach on WordNet.
  2. Formulating Consistency Constraints: They define two main types of constraints:
  3. Cons.ws: Predicts that the chosen sense for a word should be close to the entity's centroid concept.
  4. Cons.cr: Predicts that a mention with a given sense should be assigned to an entity whose centroid concept is compatible with that sense.
  5. Using Markov Logic Networks: These constraints are implemented as rules in MLNs, a formalism that allows for "violable constraints" (soft rules) with associated weights, enabling statistical relational inference. They explore different rule sets (Set I, II, III) with varying approaches to distance thresholds and learned weights.

Datasets

The research utilizes a corpus of ritual descriptions from Nepal, comprising 48 texts and approximately 97,460 tokens. This corpus presents unique challenges due to domain-specific terminology.

  • Training/Development/Test Data: The paper describes the creation of manually annotated data for WSD and CR to serve as development and test sets. Table 5 provides an overview of the size of these sets in terms of tokens, types, sense ambiguity, mentions, and chains.
  • Existing Resources: They leverage WordNet as a sense inventory for WSD and FrameNet for semantic role labeling. They also mention using the Wall Street Journal (WSJ) corpus for initial domain adaptation of NLP tools. A Digital Corpus of Sanskrit (DCS) was used to acquire domain-specific sense information for WSD.

Key Insights and Ideas

  • Inter-level consistency is crucial: The paper strongly advocates for looking beyond individual annotation layers to find errors. The interaction between WSD and CR is a rich source of potential inconsistencies.
  • Discourse entities have semantic coherence: The assumption that mentions within a discourse entity should share related senses is a powerful principle for error detection.
  • Markov Logic Networks are well-suited for this task: MLNs provide a flexible framework to express these complex, probabilistic relationships between different annotation levels.
  • Domain adaptation is essential: Standard NLP tools perform poorly on specialized corpora like ritual texts, highlighting the need for adaptation techniques.

Key Results

The paper reports on error detection performance for WSD and CR:

  • Coreference Resolution (CR): The MLN-based approach shows promising results for error detection. With the "Set I" rule set (using learned weights), they achieve 69.6% precision and a 77.1% f-score. This level of performance is considered potentially cost-effective for supporting manual annotation control.
  • Word Sense Disambiguation (WSD): Error detection for WSD yielded devastatingly low precision. This suggests that the current constraints and MLN formulation are not yet sufficient to reliably identify WSD errors.
  • Impact of Domain Adaptation: Domain adaptation techniques improved the performance of individual WSD and CR systems significantly before applying the MLN consistency checks. For example, domain-adapted WSD closely matched the MFS baseline, and domain-adapted CR showed substantial gains in precision and recall.

Limitations and Future Work

  • Small Evaluation Datasets: The authors acknowledge that their evaluation is based on small datasets, limiting the statistical significance of the results. Larger datasets are needed for more robust conclusions.
  • Noisy Centroids: The current centroids are computed on top of automatic CR and WSD annotations, which are themselves noisy. This limits the ability to identify novel or more homogeneous discourse entities.
  • Separate Rule Sets: The current experiments use distinct rule sets for CR and WSD error detection. Future work will investigate joint processing of these constraints to fully exploit MLN's power.
  • Integrating Centroid Computation: The computation of centroids is not yet integrated into the MLN inference process, meaning changes in sense or mention assignment don't directly influence centroid calculation in real-time.
  • Improving WSD Error Detection: Significant improvements are needed to achieve realistic performance levels for WSD error detection. This may involve more refined constraint sets, incorporating surface information, and exploring models similar to Set III that are less dependent on distance threshold optimization.
  • One-Sense-Per-Discourse Hypothesis: Explicitly formulating and integrating this hypothesis into the constraints is a promising direction.

Summary updated

Abstract

This contribution investigates novel techniques for error detection in automatic semantic annotations, as an attempt to reconcile error-prone NLP processing with high quality standards required for empirical research in Digital Humanities. We demonstrate the state-of-the-art performance of semantic NLP systems on a corpus of ritual texts and report performance gains we obtain using domain adaptation techniques. Our main contribution is to explore new techniques for annotation consistency control, as an attempt to reconcile error-prone NLP processing with high quality requirements. The novelty of our approach lies in its attempt to leverage multi-level semantic annotations by defining interaction constraints between local word-level semantic annotations and global discourse-level annotations. These constraints are defined using Markov Logic Networks, a logical formalism for statistical relational inference that allows for violable constraints. We report first results.
BibTeX
@inproceedings{frank2012semantic,
  author = {Anette Frank and Thomas Bögel and Oliver Hellwig and Nils Reiter},
  title = {{Semantic Annotation for the Digital Humanities}},
  booktitle = {Linguistic},
  year = {2012},
  doi = {10.33011/lilt.v7i.1275},
}
Figure