TL;DR
This paper tackles the challenge of creating semantic annotation schemes that work across many languages. They propose a "lattice-like" architecture for organizing semantic categories, which allows for both language-specific nuances and universal generalizations, and demonstrate its effectiveness with a pilot annotation experiment.
What Problem Does This Paper Address?
Developing computational linguistic tools often relies on annotating text with semantic information. However, creating annotation schemes that are applicable across different languages is a significant hurdle. Languages naturally carve up conceptual space in diverse ways, leading to issues when trying to apply a single set of labels. This paper focuses on four key challenges: defining base annotation labels, accommodating languages with coarser or finer-grained distinctions than the proposed scheme, and handling distinctions that "cross-cut" the established categories. The goal is to create a system that is both universally applicable and sensitive to linguistic diversity.
Methods
The authors propose a lattice-like architecture for organizing semantic annotation categories. Instead of a strict hierarchy, this structure allows categories to be organized in overlapping groupings across different levels. The "base level" contains categories that are typologically common across languages. Higher and lower levels provide coarser and finer-grained distinctions, respectively. This structure is designed to be flexible, allowing annotators to choose labels from different levels of the lattice based on the language and the specific context. They also propose guidelines for annotators on how to select labels from this lattice, particularly when dealing with ambiguous or cross-cutting categories.
Datasets
The paper describes a pilot annotation experiment using thirty-six English sentences from the STREUSLE corpus, which express spatial figure-ground relations. These sentences were translated into Dutch, Czech, and Korean. The PPs (prepositional phrases) from these sentences served as the annotation targets.
Key Insights and Ideas
- Lattice over Hierarchy: A strict hierarchy struggles with languages that have "cross-cutting" categories. A lattice, with its overlapping groupings, offers a more flexible way to represent these complex relationships.
- Typological Motivation is Key: Basing the "base level" categories on distinctions common across many languages makes the scheme broadly applicable.
- Flexibility for Annotators: Allowing annotators to choose labels from different levels of the lattice accommodates both language-specific granularity and the need for cross-linguistic comparability.
- Handling Cross-Cutting Categories: The proposed lattice architecture offers a way to manage categories that don't neatly fit into a single predefined slot, by allowing annotators to choose higher-level, more general labels when necessary.
- Compatibility vs. Identity: The pilot study highlights that "compatibility" of annotations (where labels can be traced back to equivalent concepts in the lattice) is a more robust measure of cross-linguistic portability than strict "identity" of labels.
Key Results
The pilot annotation experiment showed promising results. Pairwise agreement between annotators for different languages was high, with raw proportions ranging from 82% to 93%. Cohen's Kappa scores were also acceptable (0.64-0.86). Crucially, pairwise compatibility proportions were, on average, 7% higher than identity scores, and compatibility Kappa scores were 0.15 higher. This suggests that the lattice architecture effectively allows for abstracting away from language-specific subdivisions of semantic space while maintaining comparability. For example, an "attachment" annotation in Dutch (which has a specific preposition for attachment) was compatible with a "non-containment" annotation in English, demonstrating the lattice's ability to bridge differences in granularity.
Limitations and Future Work
The authors acknowledge that a reviewer pointed out the difficulty of assessing the improvement of their lattice over a flat annotation scheme. Due to time constraints, they couldn't conduct a new experiment with a control group. Future work could involve comparing their lattice approach to flat schemes on new corpora. They also note that some disagreements in the pilot study stemmed from genuinely different conceptualizations of spatial relations or from language-specific conventionalized construals, which are harder to resolve with annotation schemes alone. They suggest that further research is needed to see if these types of disagreements are common across semantic domains.
Summary updated