TL;DR
Researchers have introduced NLI-PT, the first dedicated dataset for Native Language Identification (NLI) in Portuguese. By unifying and annotating nearly 1,900 essays from learners across 15 different native languages, this resource enables new research into how a speaker's mother tongue influences their acquisition of Portuguese.
What Problem Does This Paper Address?
Native Language Identification (NLI) is the task of determining an author's first language (L1) based solely on their writing in a second language (L2). It is a powerful tool for Second Language Acquisition (SLA) research, as it helps identify linguistic transfer patterns—the specific ways a native English speaker might struggle with Portuguese grammar compared to a native Chinese speaker.
While NLI has seen significant progress in English (thanks to shared tasks and large corpora), other languages have lagged behind. Prior to this work, Portuguese learner data was scattered across several small, inconsistently formatted corpora, most of which lacked the deep linguistic annotations (like syntax trees) required for sophisticated NLI models. The authors address this by creating a unified, multi-level annotated dataset specifically designed for NLI.
Methods
The authors didn't just collect new data; they performed a "harmonization" of three existing Portuguese learner corpora: COPLE2, the Leiria corpus, and PEAPL2.
Key steps in their pipeline included:
- Format Unification: Converting diverse formats (XML and various TXT styles) into a clean, unified text format.
- Preprocessing: Removing transcription-specific annotations (like teacher corrections or metadata) to provide a "clean" version of the student's original output.
- Multi-level Annotation: They applied several NLP toolkits to add layers of information beyond raw text:
- POS Tagging: Using the LX Parser for basic parts-of-speech and Freeling for fine-grained morphological features.
- Syntactic Parsing: Generating both constituency and dependency parses using the LX Parser and the DepPattern toolkit.
To demonstrate the dataset's utility, they implemented a baseline NLI system using a linear Support Vector Machine (SVM) with a simple bag-of-words model.
Datasets
The resulting NLI-PT dataset is a significant consolidation of resources. As shown in the distribution table below, it brings together 1,868 texts totaling over 380,000 tokens.
The dataset covers 15 native languages, including Chinese, English, Spanish, German, Russian, and others. It also categorizes texts by proficiency levels based on the Common European Framework of Reference for Languages (CEFR), ranging from A1 (beginner) to C1 (advanced).
Key Insights and Ideas
- The Challenge of Topic Bias: One of the most interesting findings involves the distribution of essay topics. In NLI, models often "cheat" by learning that certain L1 groups tend to write about specific topics, rather than learning linguistic patterns. The authors found that in NLI-PT, just three topics account for nearly half of all texts, while 72% of topics have very few examples.
- Linguistic Transfer as a Feature: By providing constituency and dependency parses, the authors enable researchers to look past simple vocabulary (which is prone to topic bias) and investigate structural "fingerprints" left by an author's native language.
- Educational Utility: Beyond NLI, the authors highlight that this dataset is a goldmine for developing Portuguese-specific educational tools, such as automated essay scoring, L1-tailored grammar checkers, and Computer-aided Language Learning (CALL) software.
Key Results
The authors established a baseline for Portuguese NLI by selecting the five most frequent L1s in the dataset: Chinese, English, German, Italian, and Spanish.
Using a linear SVM with a bag-of-words model and stratified 10-fold cross-validation, they achieved 70% accuracy. While impressive for a simple baseline, the authors caution that this result likely reflects some degree of topic bias, given the uneven distribution of prompts across the L1 groups.
Limitations and Future Work
The primary limitation acknowledged is the lack of topic balance. Because the source corpora were collected for different purposes, the L1 groups are not evenly distributed across the 148 topics. This makes it difficult to disentangle "language patterns" from "topic patterns."
Future work aims to:
- Expand the dataset: Incorporating data from the CAL2 corpus to increase the sample size.
- Mitigate Bias: Experimenting with syntactic features (function words, POS n-grams, and dependency relations) which are generally more robust against topic bias than bag-of-words models.
- Refine Annotations: Continuing to improve the quality of the linguistic parses provided with the dataset.
Summary updated