Proceedings of the Third Workshop on Computational Typology and Multilingual NLP

Ekaterina Vylomova, Elizabeth Salesky, Sabrina Mielke, Gabriella Lapesa, Ritesh Kumar, Harald Hammarström, Ivan Vulić, Anna Korhonen, Roi Reichart, Edoardo Maria Ponti, Ryan CotterellNAACL 2021

TL;DR

This paper presents the proceedings of SIGTYP 2021, the third workshop focused on computational typology and its integration with multilingual Natural Language Processing (NLP). It showcases research on leveraging linguistic typology to improve NLP tasks, developing typologically-informed models, and creating resources for under-resourced languages, highlighting the growing intersection of linguistic theory and practical NLP applications.

What Problem Does This Paper Address?

The field of Natural Language Processing (NLP) has made significant strides, but many state-of-the-art models still struggle with the diversity of human languages. This is particularly true for low-resource languages, where data scarcity hinders model development. Furthermore, traditional NLP models often treat languages as independent entities, neglecting the rich patterns of linguistic variation and shared features across languages that typology studies. This workshop addresses the challenge of building more robust, adaptable, and linguistically informed NLP systems by bridging the gap between computational linguistics and linguistic typology. It aims to explore how typological knowledge can enhance NLP tasks, leading to better performance, especially for diverse and under-represented languages.

Methods

The papers presented in these proceedings cover a wide range of methodologies, reflecting the interdisciplinary nature of the workshop. Common themes include:

  • Machine Learning Models: Various machine learning architectures are employed, including Graph Convolutional Networks (GCNs), Recurrent Neural Networks (RNNs) like LSTMs, Convolutional Neural Networks (CNNs), and Transformer-based models.
  • Cross-lingual Transfer Learning: Techniques for transferring knowledge from high-resource languages to low-resource languages are explored, often involving unsupervised or semi-supervised methods.
  • Linguistic Typology Integration: Several papers focus on explicitly incorporating typological features into NLP models. This can involve using typological features as input, designing models that are sensitive to typological variation, or developing new metrics inspired by typological concepts.
  • Data Augmentation and Generation: Methods for generating synthetic data or augmenting existing datasets are used to address data scarcity.
  • Information Theory: Concepts from information theory are applied to analyze linguistic phenomena, such as morphological fusion and orthographic transparency.
  • Probabilistic Graphical Models: These models are proposed as a framework for studying linguistic typology and predicting feature values.

Datasets

The proceedings mention or utilize a variety of datasets, with a notable emphasis on:

  • Existing Corpora: Standard NLP corpora are used, but often with a focus on specific languages or tasks.
  • Multilingual Datasets: Datasets covering multiple languages are crucial for cross-lingual research.
  • Newly Introduced Benchmarks:
    • Let-Mi: An Arabic Levantine Twitter dataset for misogynistic language, aiming to address the lack of benchmark datasets for this specific type of toxic content in Arabic. It includes tweets annotated for neutrality or one of seven misogyny categories.
  • Cross-lingual Dataset for Intent Classification and Slot Detection: A novel dataset for digital assistant tasks, comprising 13 languages, introduced to facilitate research in this area.

Key Insights and Ideas

  • Typology as a Bridge: Linguistic typology is not just an academic pursuit but a valuable source of knowledge that can significantly improve NLP model performance, especially for low-resource languages.
  • Beyond Surface Patterns: Moving beyond surface-level linguistic features and incorporating deeper typological understanding can lead to more robust and generalizable NLP systems.
  • The Power of Pseudo-Labels: Unsupervised methods, particularly those leveraging pre-trained models and pseudo-labeling, show promise in overcoming data scarcity for cross-lingual tasks.
  • Quantifying Linguistic Properties: Novel methods are being developed to quantify abstract linguistic properties, such as morphological fusion and orthographic transparency, using information-theoretic approaches.
  • Figurative Language Generation: The generation of creative and novel figurative language, like metaphors and similes, is an emerging area of research, pushing the boundaries of text generation.
  • The Importance of Context: Understanding the context of language use, such as in social media or spoken interactions, is critical for developing effective NLP applications.

Key Results

  • Improved Performance: Several papers report improvements in NLP tasks (e.g., sentiment analysis, language identification, news classification) by incorporating typological features or using typologically-informed methods.
  • Competitive Unsupervised Models: Unsupervised models for cross-lingual transfer tasks have demonstrated performance competitive with supervised counterparts, sometimes within a few percentage points of F1 score.
  • Novel Dataset Contributions: The introduction of datasets like Let-Mi provides crucial resources for under-researched areas and languages, enabling further development and evaluation of NLP models.
  • Insights into Language Variation: Research using information theory and probabilistic models offers new ways to understand and quantify linguistic variation and relationships between languages.

Limitations and Future Work

  • Data Scarcity Remains a Challenge: Despite advancements in unsupervised methods, data scarcity for many languages continues to be a significant hurdle.
  • Generalizability of Typological Features: While typological features show promise, their effective integration and generalizability across diverse NLP tasks and language families require further investigation.
  • Interpretability of Complex Models: Understanding why complex models, especially those incorporating typological knowledge, perform well remains an area for future research.
  • Expanding Typological Coverage: Future work will likely focus on incorporating a wider range of typological features and exploring their impact on more NLP tasks.
  • Developing More Sophisticated Cross-lingual Methods: Continued research into unsupervised and semi-supervised cross-lingual transfer learning is expected, aiming to further close the gap with supervised approaches.
  • Creative Text Generation: Advancements in generating figurative language and other creative text forms are anticipated.

Summary updated

BibTeX
@inproceedings{vylomova2021proceedings,
  author = {Ekaterina Vylomova and Elizabeth Salesky and Sabrina Mielke and Gabriella Lapesa and Ritesh Kumar and Harald Hammarström and Ivan Vulić and Anna Korhonen and Roi Reichart and Edoardo Maria Ponti and Ryan Cotterell},
  title = {{Proceedings of the Third Workshop on Computational Typology and Multilingual NLP}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure