The Role of Emotions in Native Language Identification

Ilia Markov, Vivi Nastase, Carlo Strapparava, Grigori SidorovEMNLP 2018

TL;DR

This paper investigates whether emotional expression in writing can reveal an author's native language (L1), even when they are writing in a second language (L2). They found that incorporating emotion-based features significantly improves the accuracy of identifying an author's L1, suggesting that emotional nuances are indeed transferred from L1 to L2.

What Problem Does This Paper Address?

Native Language Identification (NLI) is the task of determining an author's L1 based on their writing in an L2. This has practical applications in areas like security, marketing, and education. While NLI research has explored various linguistic features (like word and character n-grams, part-of-speech tags), the role of emotions has been less explored. The authors hypothesize that emotions, being culturally and linguistically specific, might be a dimension of language that "leaks" from an author's L1 into their L2 writing, thus serving as a signal for NLI. They aim to quantify this impact and understand how emotion features contribute to NLI accuracy.

Methods

The core of the methodology involves using machine learning classifiers (specifically Support Vector Machines with an OvR strategy) to predict an author's L1. The key innovation lies in the features used:

  • Baseline Features: Standard NLI features like Part-of-Speech (POS) tag n-grams and function word (FW) n-grams were used as a baseline. These are considered robust against topic bias.
  • Emotion Features:
    • Emotion Polarity (emoP): This feature encodes binary associations from the NRC Word-Emotion Association Lexicon. For each word, it captures its association with 8 emotions (anger, fear, anticipation, trust, surprise, sadness, joy, disgust) and 2 sentiments (negative, positive). This is represented as a categorical feature.
  • Emotion Load (emoL): This captures the overall emotional intensity of a text. It includes:
  • A binary feature indicating whether an essay has a high or low emotional load compared to the dataset average.
  • A numeric feature representing the ratio of emotion words in an essay.
  • 10 numeric features representing the ratio of each specific emotion/sentiment.

These emotion features were then combined with the baseline POS and POS & FW features to assess their impact.

Datasets

The study utilized two established datasets commonly used in NLI research:

  • TOEFL11: This corpus contains 1,100 essays in English written by non-native speakers from 11 different L1s. It includes information on author proficiency levels (low, medium, high).
  • ICLE (International Corpus of Learner English): This dataset consists of essays from highly proficient non-native English speakers. The study used a 7-language subset of this corpus.

Key Insights and Ideas

  • Emotion as a "Leak": The central finding is that emotional expression in L2 writing is influenced by an author's L1. This suggests that cultural and linguistic habits surrounding emotion expression are not entirely shed when learning a new language.
  • Emotion Features are Informative: Even simple emotion-based features, like the presence of emotion words, significantly improve NLI accuracy compared to random word features of the same size. This indicates that emotion words carry more discriminative power than random vocabulary.
  • Cultural Specificity of Emotion: The study supports the hypothesis that emotion usage is culture-specific. Different L1s lead to different patterns in how emotions are expressed, making them useful for identifying the author's origin.
  • Proficiency Matters: The impact of emotion features is more pronounced for authors with medium and high proficiency levels. This is attributed to their greater ability to choose words that accurately convey nuanced emotions, making these subtle L1 influences more detectable.
  • Generalization Across Topics: While some topics might naturally elicit more emotional responses, emotion-based features generally improve NLI across various topics, though the improvement might be less statistically significant for topics like "ideas vs. facts" or "education."

Key Results

  • Significant Accuracy Gains: Adding emotion-based features (specifically emoP) to a baseline POS tag n-gram model resulted in accuracy improvements of over 10% on both TOEFL11 and ICLE datasets.
  • Combined Features: When combined with POS and Function Word (FW) n-grams, emotion features (emoP) also led to significant accuracy improvements, reaching over 3% on both datasets.
  • Emotion Load Features Help: The emoL features, which capture the overall emotional intensity, also improved results in most settings, supporting the idea that some cultures might use a broader or narrower emotional vocabulary.
  • Proficiency Impact: For the TOEFL11 dataset, accuracy improvements from emotion-based features were notably higher for medium and high proficiency levels compared to low proficiency levels.

Limitations and Future Work

  • Controlled Environment: The study was conducted on learner corpora, which might represent a more "controlled environment" with a tendency towards neutral writing styles. The authors suggest that the detected effect of emotion might be even stronger in less constrained writing contexts.
  • Fine-grained emoL: While the emoL features showed promise, the authors suggest that more fine-grained analysis of emotion load features could potentially yield further improvements.
  • Topic Dependency: While emotion features generally helped across topics, the improvement was less consistent for certain topics. Further investigation into topic-specific emotion usage could be beneficial.
  • Implicit Task: The essays were not explicitly tasked with expressing emotions, yet emotion signals were detected. Future work could explore how explicit emotion prompts might influence these findings.

Summary updated

BibTeX
@inproceedings{markov2018role,
  author = {Ilia Markov and Vivi Nastase and Carlo Strapparava and Grigori Sidorov},
  title = {{The Role of Emotions in Native Language Identification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
}
Figure