Proceedings of the 13th Linguistic Annotation Workshop

Annemarie Friedrich, Deniz Zeyrek, Jet HoekACL 2019

TL;DR

This document presents the proceedings of the 13th Linguistic Annotation Workshop (LAW XIII), a forum for researchers to share advancements in corpus annotation, manipulation, and exploitation. The workshop featured 28 accepted papers, covering a wide range of linguistic annotation tasks and challenges, with a special theme on "Marking of information quality in discourse."

What Problem Does This Paper Address?

The Linguistic Annotation Workshop (LAW) series aims to address the ongoing need for robust and standardized methods in linguistic annotation. This includes developing tools and frameworks for annotated language resources, fostering harmonization and interoperability, and building consensus on crucial issues in corpus annotation. The specific theme of LAW XIII, "Marking of information quality in discourse," highlights the challenge of annotating how speakers and writers express assessments, including explicit and implicit markers of belief, opinion, and appraisal. This is crucial for understanding nuanced communication, such as a politician's claims or a reporter's stance.

Methods

The proceedings showcase a diverse array of methods employed in linguistic annotation. These include:

  • Crowdsourcing: Techniques for gathering annotations from a large number of non-expert annotators, as seen in "Crowdsourced Hedge Term Disambiguation" and "Crowdsourcing Discourse Relation Annotations by a Two-Step Connective Insertion Task."
  • Annotation Scheme Development: The creation of new annotation guidelines and frameworks for specific linguistic phenomena, such as "A New Annotation Scheme for the Sejong Part-of-speech Tagged Corpus" and "Toward Dialogue Modeling: A Semantic Annotation Scheme for Questions and Answers."
  • Tool Development: The creation of software tools to facilitate annotation, exemplified by "CCGweb: a New Annotation Tool and a First Quadrilingual CCG Treebank" and "An Online Annotation Assistant for Argument Schemes."
  • Corpus Creation and Annotation: The development of new annotated corpora for various languages and tasks, including "The Making of the Litkey Corpus" and "A Turkish Dataset for Gender Identification of Twitter Users."
  • Evaluation Methodologies: Approaches to assess the quality and consistency of annotations, such as "Comparative Judgments are More Consistent than Binary Classification for Labelling Word Complexity" and "Continuous Quality Control and Advanced Text Segment Annotation with WAT-SL 2.0."
  • Cross-Linguistic Studies: Investigations into the applicability and adaptation of annotation schemes across different languages, as seen in "Annotating Information Structure in Italian: Characteristics and Cross-Linguistic Applicability of a QUD-Based Approach."

Datasets

The proceedings describe the use and creation of numerous datasets. While a comprehensive list is beyond the scope of this summary, notable examples include:

  • Litkey Corpus: A richly annotated longitudinal corpus of German texts written by primary school children.
  • Materials Science Procedural Text Corpus: Annotated with shallow semantic structures for materials synthesis procedures.
  • Sejong Part-of-speech Tagged Corpus: A corpus for which a new annotation scheme was developed.
  • Hindi-English Code-Mixed Tweets: A dataset specifically created for semantic role labeling.
  • Turkish Treebank: Efforts to unify and construct Turkish treebanking resources.
  • Latin Linguistic Resources: Used for harmonizing lemmatization strategies.
  • Hungarian Corpora: Used for creating corpora with semantic role labels and for developing pipelines.
  • Brazilian Portuguese Corpus: Aimed at creating a General Abstract Meaning Representation corpus.

Several papers also introduce new benchmarks or focus on adapting existing ones for specific tasks and languages.

Key Insights and Ideas

  • The growing importance of discourse-level annotation: The special theme highlights the increasing need to capture how information quality is conveyed in discourse, moving beyond sentence-level semantics.
  • Crowdsourcing as a viable annotation strategy: Multiple papers demonstrate the effectiveness of crowdsourcing for tasks like hedge term disambiguation and discourse relation annotation, suggesting its scalability for large-scale annotation projects.
  • The challenge of harmonizing annotation practices: The paper on harmonizing lemmatization strategies for Latin underscores the persistent difficulty in achieving consistency across different annotation approaches, even for well-established linguistic phenomena.
  • The utility of specialized annotation tools: The development of tools like CCGweb and an online assistant for argument schemes indicates a trend towards more user-friendly and efficient annotation processes.
  • The need for cross-linguistic annotation resources: The creation of datasets and adaptation of schemes for languages like Hindi, Turkish, and Portuguese addresses the gap in resources for non-English languages.
  • The role of back-translation in corpus generation: One paper explores back-translation as a strategy for creating corpora for non-English tasks, offering a potential solution for data scarcity.

Key Results

The proceedings present a wide range of results, often specific to the individual papers. However, some general trends and findings can be inferred:

  • Successful development of new annotation schemes and tools: Several papers report on the successful creation and evaluation of novel annotation frameworks and software.
  • Demonstrated effectiveness of crowdsourcing: Papers utilizing crowdsourcing often show promising results in terms of annotation quality and coverage, though specific metrics vary.
  • Progress in building annotated corpora: The workshop showcases the ongoing effort to expand the availability of annotated data for various languages and linguistic phenomena.
  • Insights into annotation consistency and quality control: Research on comparative judgments and continuous quality control offers methods to improve the reliability of annotations.

Limitations and Future Work

  • Scalability of annotation: While crowdsourcing shows promise, challenges related to annotator training, quality control, and managing large-scale annotation projects remain.
  • Cross-linguistic transferability: Adapting annotation schemes and tools across languages often requires significant effort and may not always yield equivalent results.
  • Annotation of implicit information: Capturing subtle nuances like information quality in discourse presents a significant challenge that requires further research.
  • Development of robust evaluation metrics: As annotation tasks become more complex, there is a continuous need for sophisticated evaluation metrics that accurately reflect linguistic phenomena.
  • Future work is likely to focus on developing more automated and semi-automated annotation methods, expanding resources for under-resourced languages, and refining annotation schemes for complex discourse phenomena.

Summary updated

BibTeX
@inproceedings{friedrich2019proceedings,
  author = {Annemarie Friedrich and Deniz Zeyrek and Jet Hoek},
  title = {{Proceedings of the 13th Linguistic Annotation Workshop}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure