Punctuation as Native Language Interference

Ilia Markov, Vivi Nastase, Carlo StrapparavaCOLING 2018

TL;DR

This paper investigates whether punctuation patterns in English writing can reveal an author's native language (L1). The authors demonstrate that punctuation is a surprisingly robust indicator of L1, even for highly proficient English learners, and can significantly improve the accuracy of Native Language Identification (NLI) systems.

What Problem Does This Paper Address?

Native Language Identification (NLI) is the task of determining a person's native language based on their writing in a second language (L2). This has applications in areas like security, marketing, and education. Previous NLI research has explored various linguistic features like character n-grams, lexical choice, grammar, and spelling errors. However, the role of punctuation, a seemingly fundamental aspect of written language that varies across languages, has been largely overlooked. This paper aims to fill that gap by systematically evaluating the impact of punctuation on NLI.

Methods

The authors employ a machine learning approach, specifically Support Vector Machines (SVM) with an OvR (one-vs-rest) multi-class strategy. They represent essays using two primary feature sets:

  • Part-of-Speech (POS) n-grams: These capture the morpho-syntactic patterns of a text. The authors use POS tags from the Penn Treebank tagset and consider n-grams of size 1, 2, and 3.
  • Punctuation Marks (PMs): To incorporate punctuation, they are added to the POS n-gram features. The paper notes that while punctuation marks alone can achieve some level of classification, their true power lies in their integration with other features.

They conduct several experimental setups:

  • Multi-class classification: Directly classifying the L1 of an author.
  • 2-step classification: A coarse classification into language families/geographical groups, followed by a fine-grained classification within each group. This helps explore commonalities and specificities within language families.
  • Proficiency-level classification: Analyzing how punctuation's influence changes with the learner's English proficiency (low, medium, high).
  • Cross-topic and cross-corpus classification: Testing the robustness of punctuation features across different writing topics and entirely different datasets to assess their generalizability and resistance to overfitting.

Datasets

The study utilizes two commonly used NLI datasets:

  • TOEFL11: This corpus contains 1,100 English essays from learners of 11 different native languages, with information on proficiency levels and writing prompts.
  • ICLEv2: This dataset consists of essays from highly proficient non-native college-level students of English, with a 7-language subset used in this study.

No new benchmarks were introduced; the focus was on evaluating existing datasets.

Key Insights and Ideas

  • Punctuation is a surprisingly strong L1 indicator: Even for highly proficient learners, native language punctuation habits persist and significantly aid in identifying their L1.
  • Punctuation complements other features: While POS n-grams are useful, adding punctuation features consistently and substantially improves NLI accuracy across various experimental settings.
  • Language families and individual languages matter: The 2-step classification reveals both common punctuation patterns within language families and unique styles specific to individual languages.
  • Proficiency doesn't erase L1 influence: Contrary to expectations, the improvement from punctuation features increases with higher proficiency levels, suggesting learners maintain their L1 punctuation style.
  • Punctuation is robust and less prone to overfitting: Unlike word n-grams, which can capture topic-specific information and overfit, punctuation features are more abstract and perform reliably across different topics and corpora.

Key Results

  • Significant Accuracy Improvements: Adding punctuation features to POS n-grams consistently boosted accuracy. For example, in the multi-class classification on TOEFL11, POS 1-3-grams with punctuation achieved 43.11% accuracy, a substantial improvement over 32.40% without punctuation. Similar gains were observed across all experimental settings.
  • High Proficiency Learners Still Show L1 Influence: In the balanced TOEFL11 dataset, for high-proficiency learners, adding punctuation improved accuracy from 31.31% to 39.39% (1-step) and from 31.93% to 37.25% (1-step), demonstrating persistent L1 interference.
  • Robustness Across Topics and Corpora: In cross-topic experiments, punctuation features maintained their performance gains. In cross-corpus experiments (TOEFL to ICLE), POS 1-3-grams with PMs achieved 54.50% accuracy compared to 43.27% without, a significant improvement.

Limitations and Future Work

  • Statistical Significance: While improvements are noted, the paper mentions that some specific improvements might not be statistically significant (e.g., in cross-corpus experiments).
  • Feature Representation: The paper notes that punctuation marks used in isolation might not fully capture their usage and role in a sentence. Future work could explore more sophisticated ways to encode punctuation's contextual function.
  • Deeper Linguistic Exploration: The authors suggest that their findings could support deeper linguistic exploration into the specific punctuation phenomena that are characteristic of certain languages and interfere in L2 writing.
  • Broader Language Coverage: While the study uses established datasets, expanding to a wider range of languages and language families could provide a more comprehensive understanding of punctuation interference.

Summary updated

BibTeX
@inproceedings{markov2018punctuation,
  author = {Ilia Markov and Vivi Nastase and Carlo Strapparava},
  title = {{Punctuation as Native Language Interference}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure