NLP for Language Documentation: Two Reasons for the Gap between Theory and Practice

Luke Gessler, Katharina von der WenseNAACL 2024

TL;DR

Despite the potential of Natural Language Processing (NLP) to aid in the preservation of endangered languages, its adoption in language documentation (LD) remains surprisingly low. This paper investigates two key reasons for this gap: the scarcity of interdisciplinary training in university curricula and the low rate of interdisciplinary collaboration in research publications.

What Problem Does This Paper Address?

The world is facing a critical loss of linguistic diversity, with roughly half of the planet's 7,000 languages considered endangered. Language documentation (LD) is a labor-intensive process crucial for preserving these languages, but the demand for skilled documentarians far outstrips supply. While NLP and computational linguistics (CL) have developed tools that could significantly automate parts of this process (e.g., automatic transcription), these tools are not widely adopted in real-world LD projects. This disconnect represents a missed opportunity to accelerate language preservation efforts. The authors aim to empirically investigate two proposed explanations for this gap: the lack of interdisciplinary education and collaboration between NLP researchers and documentary linguists.

Methods

The authors tackle their research questions by analyzing two primary data sources: university curricula and academic publications.

For university curricula, they examined the top 25 universities in five countries (USA, Germany, Brazil, Mexico, and Colombia). They identified graduate programs in computer science (CS) and linguistics (Ling) and then determined if these programs offered courses relevant to NLP/CL or LD. Specific criteria were established for what constituted an "NLP/CL course" (e.g., dedicated NLP focus, not just an AI intro) and an "LD course" (e.g., practical language documentation methods).

For academic publications, they collected a large number of papers related to NLP for LD. They defined "relevance" based on whether the work's core contribution could aid documentation projects, operationalized by using small datasets (relative to high-resource languages) and focusing on tasks directly relevant to LD. They also annotated papers for:

  • Country: The primary country of author affiliations.
  • Documentation as Purpose (DaP): Whether LD was explicitly mentioned as a motivation.
  • Performance of Documentation (POD): Whether novel primary language data was collected as part of the work.
  • Interdisciplinarity (Int): Whether the author list included at least one NLP researcher and one documentary linguist, assessed by publication venue.

They utilized two resources for publication data: the ACL Anthology (specifically AmericasNLP and ComputEL workshops) and Semantic Scholar, employing a keyword-based search strategy.

Datasets

The paper does not introduce new datasets for training or evaluation. Instead, it analyzes existing data from:

  • University Curricula: Data was collected by examining the course catalogs and program descriptions of the top 25 universities in the USA, Germany, Brazil, Mexico, and Colombia.
  • Academic Publications: Data was sourced from the ACL Anthology (AmericasNLP and ComputEL workshops) and Semantic Scholar. The authors performed a manual annotation of relevant papers based on specific criteria.

Key Insights and Ideas

  • The "Theory-Practice Gap" is Real and Quantifiable: The paper moves beyond anecdotal evidence to empirically demonstrate the low integration of NLP tools in language documentation.
  • Education is a Bottleneck: The scarcity of graduate programs that bridge NLP and LD is a significant barrier. Students are often siloed in their respective fields, missing opportunities for cross-pollination.
  • Interdisciplinary Collaboration is Low, but Growing (in some regions): While overall interdisciplinary collaboration in publications is not high, the US shows a comparatively higher rate, potentially linked to its more developed interdisciplinary educational offerings.
  • The US Leads in Interdisciplinary Awareness: The US has a higher proportion of universities offering interdisciplinary courses and a higher rate of interdisciplinary publications in NLP for LD, suggesting a greater awareness of LD challenges within its NLP community.
  • Specialized Workshops Help: The paper implicitly supports the idea that focused workshops (like LTLDR) are valuable for fostering interdisciplinary connections.

Key Results

  • University Curricula:
    • None of the 93 examined CS departments offered an LD-related course.
  • Of 56 linguistics departments, only 22 offered an NLP/CL course, and just 10 offered both an NLP/CL and an LD course.
  • The US had a higher rate of universities offering both NLP/CL and LD courses compared to other countries examined.
  • Publications:
    • For Semantic Scholar data, less than half of relevant papers (as low as 13%) were truly interdisciplinary.
  • For the ACL Anthology data, the US had significantly more representation and a higher percentage of works naming LD as an explicit goal.
  • Fewer than 10% of German papers cited documentation as a purpose or had an interdisciplinary team.
  • Due to limited data for Latin American countries, a definitive answer to RQ3 (connection between curricula and publications) could not be provided, but the US trend suggested a positive correlation.

Limitations and Future Work

  • Data Quantity: The study's findings are limited by the quantity of data collected, particularly for Latin American countries, making it difficult to draw broad conclusions for those regions. The selection of only 25 universities per country also limits generalizability.
  • Sampling Bias: The methods used to collect publication data (ACL Anthology and Semantic Scholar) are acknowledged to potentially introduce sampling bias.
  • Future Work: The authors suggest the need for more research to investigate other potential factors influencing adoption, such as nation-level grant programs and academic cultures. They also highlight the importance of interdisciplinary coursework and workshops in bridging the gap between NLP and LD.

Summary updated

BibTeX
@inproceedings{gessler2024nlp,
  author = {Luke Gessler and Katharina von der Wense},
  title = {{NLP for Language Documentation: Two Reasons for the Gap between Theory and Practice}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure