TL;DR
This paper introduces the first semantic tagging system for the Irish language, a crucial step for its digital preservation and advancement. By adapting the established USAS tagset and developing a novel evaluation methodology, the researchers achieve over 90% lexical coverage and nearly 80% tag accuracy, paving the way for richer NLP applications and resources for this minority language.
What Problem Does This Paper Address?
Annotated corpora are invaluable for linguistic research, machine learning, and language teaching. However, minority languages like Irish often suffer from a scarcity of such resources and underdeveloped NLP tools. This lack of semantic annotation hinders our ability to understand language usage, combat language decline, and build advanced language technologies. While part-of-speech tagging for Irish exists, semantic tagging, which assigns meaning-based categories to words, has been largely unexplored. This paper aims to bridge this gap by developing a semantic tagging pipeline for Irish.
Methods
The core of the system is built upon the USAS (UCREL Semantic Analysis System), a comprehensive and hierarchically organized semantic tagset. The authors adapted this system for Irish by:
- Lexicon Compilation: They created several lexicons by combining manually tagged data, the English USAS lexicon, and the New English-Irish Dictionary (NEID). This involved mapping English senses and subject categories to USAS tags.
- Pipeline Development: They integrated the USAS tagging capabilities into an existing Irish NLP pipeline (based on SpaCy) that handles part-of-speech tagging and lemmatization.
- Semantic Disambiguation: To address the challenge of words with multiple meanings, they implemented two disambiguation strategies:
- Document-level disambiguation: This method leverages the frequency of top-level USAS fields in a document to re-order potential tags for ambiguous words.
- Year detection: A specific component was developed to identify numbers that likely represent years and assign them the appropriate temporal tag (T1.3).
- Novel Evaluation Methodology: A key contribution is a new method for evaluating semantic tagging accuracy. This method accounts for:
- Match Value: A score between 0 and 1 for how well a predicted tag matches the ideal tag, considering partial matches and hierarchical relationships.
- Compound Tag Accuracy: A system to compare tags that represent multiple categories.
- Multi-tag Accuracy: A scoring mechanism that assigns partial credit for correctly predicted tags within a list of possible matches, weighted by their position and the total number of possible tags.
Datasets
- Golden Standard Corpus: A manually tagged corpus of 717 tokens from three diverse Irish texts (news article, Wikipedia articles on "The Wire" and "George Orwell"). This corpus was used for both lexicon building and evaluation.
- English USAS Lexicon: A pre-existing lexicon for English, adapted for Irish.
- New English-Irish Dictionary (NEID): A large dictionary containing approximately 80,000 English headwords, used to extract semantic information for Irish.
- "Core" WordNet: A list of frequently used word senses, used to enhance the USAS-en lexicon.
- Unseen Data: A selection of articles from Irish Wikipedia and news sites (1125 tokens) were used for an additional lexical coverage test.
Key Insights and Ideas
- USAS as a Viable Framework: The USAS tagset, with its hierarchical structure and extensive categories, proves to be a robust framework for semantic tagging, even for a language like Irish.
- Lexicon Combination is Key: Merging lexicons from different sources (manual, English USAS, NEID) significantly improves lexical coverage, highlighting the complementary nature of these resources.
- Disambiguation is Challenging but Crucial: While simple disambiguation methods like document-level analysis and year detection show promise, they can sometimes have a negative impact on accuracy, especially with lexicons containing highly ambiguous entries. This suggests a need for more sophisticated disambiguation techniques.
- Novel Evaluation is Essential: Traditional accuracy metrics are insufficient for semantic tagging due to the inherent ambiguity and nuanced nature of meaning. The proposed evaluation method, which considers partial matches and multi-tag assignments, provides a more realistic assessment of performance.
Key Results
- Lexical Coverage: Achieved over 90% lexical coverage when combining the USAS-en and NEID lexicons on the golden standard corpus.
- Tag Accuracy: Reached nearly 80% tag accuracy on the golden standard corpus using the combined lexicon and the full pipeline.
- Lexicon Performance: The manually built lexicon showed the highest lexical coverage (98.47%) and accuracy on the golden standard data, but this is expected due to its direct relation to the test set. The combined USAS-en + NEID lexicon offered a good balance of coverage (90.8%) and accuracy.
- Disambiguation Impact: Document-level disambiguation showed a slight negative impact on accuracy for the USAS-en lexicon, likely due to the complexity of its entries.
Limitations and Future Work
- Corpus Size and Representativeness: The "golden standard" corpus is small (717 tokens) and not representative of the full spectrum of Irish registers. Future work includes expanding this corpus.
- Lexicon Completeness: While significant progress has been made, lexicons are still incomplete, leading to lexical gaps.
- Disambiguation Sophistication: Current disambiguation methods are relatively simple. Future research will explore sentence-level and clause-level information, as well as more advanced techniques for named entity recognition and phrasal verb/modal constructions.
- ML-based Annotation: The ultimate goal is to use the semantically annotated data to train machine learning models for on-demand Irish text annotation.
- Semantic Role Labelling: The developed semantic typing pipeline is a foundational step towards semantic role labelling for Irish.
Summary updated