TL;DR
This paper introduces a novel neural network architecture for morphosyntactic tagging of morphologically complex languages, specifically for social media text. By employing an LSTM generator to predict tags character-by-character, the system effectively handles complex tag structures and previously unseen tag combinations, achieving significant improvements over traditional methods.
What Problem Does This Paper Address?
Morphosyntactic tagging, the process of assigning grammatical information (like part-of-speech, gender, number, case) to words, is crucial for understanding morphologically rich languages. However, existing data-driven models often struggle with the complexity of these tags, especially when applied to informal domains like social media. Social media text presents unique challenges: it often lacks diacritics, ignores capitalization rules, and incorporates foreign loanwords. Traditional approaches either treat complex tags as atomic units (losing sub-tag relationships) or predict each sub-tag independently (failing to model dependencies). This paper tackles the problem of accurately tagging social media text in morphologically complex languages by proposing a method that explicitly models sub-label dependencies within tags.
Methods
The core of the proposed system is a bidirectional LSTM tagger. A key innovation is the use of an LSTM generator in the output layer. Instead of predicting a complete tag at once, this generator predicts the tag character by character. This approach allows the model to:
- Handle unseen tags: By generating tags from their constituent characters, the model can construct tags it hasn't encountered during training.
- Model sub-label dependencies: The sequential nature of the generator naturally captures relationships between different parts of a tag (e.g., how gender influences case).
- Address contractions and multiple tags: The system can effectively handle cases where a single token might represent multiple grammatical units, such as contractions in Slavic languages.
The system also incorporates:
- Character-based embeddings: To handle out-of-vocabulary (OOV) words and variations in spelling.
- Pretrained FastText embeddings: Leveraging word representations trained on large text corpora to improve performance, especially with limited training data.
- Data augmentation: Applying Gaussian noise and dropout to mitigate overfitting on smaller datasets.
Datasets
The paper utilizes the following datasets:
- Twitter data: For Croatian, Serbian, and Slovenian languages, specifically collected for the VarDial 2018 shared task.
- Web data: Automatically tagged Croatian, Serbian, and Slovenian web corpora (specifically, the slWaC, bs,hr,sr wac corpora). This web data is used to augment the training data and improve tagger performance, especially for OOV words and domain adaptation.
Key Insights and Ideas
- Sub-label dependencies are crucial: The paper strongly advocates for modeling the internal structure of morphological tags, rather than treating them as monolithic units. This is shown to be more effective than independent sub-tag prediction.
- LSTM generators are powerful for complex tagging: The character-level generation approach proves effective for handling the intricate and sometimes ambiguous nature of morphosyntactic tags, especially in challenging domains.
- Social media text requires specialized handling: The authors highlight the significant differences between social media text and formal domains, emphasizing the need for robust methods that can cope with informality, OOV words, and language variations.
- Web data is a valuable resource for domain adaptation: Augmenting limited Twitter data with larger, automatically tagged web corpora significantly boosts performance.
Key Results
The proposed LSTM tagger achieved substantial improvements over a baseline HunPos trigram tagger:
- Accuracy gains: The system delivered improvements of more than 6%-points over the baseline.
- Performance across languages: The performance was reported to be quite even for all three languages (Croatian, Serbian, and Slovenian).
- Specific metrics: Table 2 shows the system achieving accuracies of 0.943 POS, 0.886 MOR, and 0.887 TAG for Croatian, 0.957 POS, 0.900 MOR, and 0.900 TAG for Serbian, and 0.946 POS, 0.884 MOR, and 0.884 TAG for Slovenian.
- Common errors: The analysis revealed confusions between noun tags and foreign word tags (Xf), and between nominative and accusative cases, particularly for inanimate nouns where overt marking is absent.
Limitations and Future Work
- Modeling foreign words: The authors acknowledge that better modeling of foreign words is needed to further improve results.
- Contextual modeling for ambiguous cases: Distinguishing between nominatives and accusatives in cases lacking overt morphological marking requires improved contextual modeling.
- Conjunctions vs. adverbs in Serbian: Accurately distinguishing between conjunctions and adverbs in Serbian, which can be challenging due to sentence structure, may require deeper sentence analysis.
- Data sparsity and web data quality: The authors suggest that some errors might stem from data sparsity or inaccuracies in the automatically tagged web data, warranting further investigation.
Summary updated