TL;DR
Large Language Models (LLMs) are surprisingly susceptible to superficial clues (like names or stereotypes) when identifying an author's native language, even when told to ignore them. This paper introduces an "agentic" approach, breaking down the task for specialized LLM "agents" to analyze different linguistic features before a final decision, leading to more robust and consistent results, especially in adversarial scenarios.
What Problem Does This Paper Address?
Native Language Identification (NLI) is the task of figuring out an author's native language (L1) based on their writing in a second language (L2). The theory is that an author's L1 leaves subtle, subconscious traces in their L2. This is useful for educational feedback and forensic linguistics.
However, current LLMs, while achieving high accuracy on NLI benchmarks, often do so by latching onto superficial contextual clues (names, locations, cultural stereotypes) rather than the deeper linguistic patterns. This makes them vulnerable to "misleading hints" – information that suggests a different L1, even if the LLM is explicitly told to ignore it. This is a problem for reliability, especially in high-stakes applications like forensic analysis, where decisions need to be grounded in objective linguistic evidence. The paper aims to improve the robustness and self-consistency of LLMs for NLI against these superficial biases.
Methods
The core of the paper's contribution is an agentic NLI pipeline. Instead of a single LLM trying to do everything, this approach breaks down the NLI task into specialized sub-tasks, each handled by a dedicated LLM "agent." This is inspired by the methodical process in forensic linguistics, where experts analyze different types of evidence in isolation before synthesizing them.
The pipeline consists of:
- Specialist Agents: These agents focus on specific linguistic categories:
- Syntax Expert: Analyzes grammatical errors (word order, subject-verb agreement, tense, etc.).
- Lexical Expert: Identifies errors in word choice (spelling, malapropisms, false cognates).
- Idiomatic Language and Translation Expert: Detects issues with multi-word expressions, idioms, and figurative language.
- Language Identification Expert (Coordinator): This final agent receives the analyses from the specialist agents and makes the overall NLI prediction. Crucially, this agent does not have direct access to the original input text, forcing it to rely solely on the synthesized linguistic evidence from the other experts.
To test robustness, the authors created adversarial task setups by appending "task hints" to the input texts. These hints were either:
- Supportive: Aligned with the true L1 of the author.
- Misleading: Suggested a different L1, either through a "Learner Signature" (name and address) or a "Stereotype Statement" (cultural generalization).
The LLM was explicitly prompted to disregard these hints.
Datasets
The experiments were conducted on two established L2 English learner corpora:
- TOEFL4: A subset of the TOEFL11 corpus, focusing on essays from native French, German, Italian, and Spanish speakers. This was chosen for its manageable size and focus on distinguishing between closely related European L1s.
- Write & Improve (W&I) Corpus 2024: A larger corpus of L2 English essays with L1 metadata. A subset was sampled to match the L1 distribution of TOEFL4 for comparability.
Key Insights and Ideas
- LLMs are easily fooled by superficial clues: Even with explicit instructions to ignore contextual information like names or stereotypes, LLMs are heavily influenced by them, leading to inconsistent predictions.
- "Disregard" instructions are not enough: Simply telling an LLM to ignore certain information doesn't guarantee it will. The models seem to have "shortcut circuits" that prioritize salient but irrelevant features.
- Agentic decomposition offers robustness: Breaking down the NLI task into specialized agents forces a more systematic, evidence-based analysis, leading to more consistent predictions even when faced with misleading information.
- Forensic linguistics as a model: The paper successfully emulates the methodical, evidence-driven approach of forensic linguists, highlighting its value for building more reliable AI systems.
- Trade-off between peak accuracy and consistency: The agentic approach might not always achieve the absolute highest accuracy in ideal conditions (compared to direct prompting), but it offers significantly greater stability and robustness against adversarial inputs.
Key Results
- Direct Prompting Vulnerability: Under misleading hint conditions, direct prompting LLMs saw a drastic drop in performance, often falling to near chance-level accuracy. For example, on the TOEFL4 dataset, performance dropped from ~95% to as low as 13.6% with misleading signatures.
- Agentic Approach Consistency: The agentic pipeline showed much more stable performance across different experimental settings. While peak accuracy on unmodified text was lower (e.g., 73.7% for Llama-3.3-70B on TOEFL4), its performance degraded much less under misleading conditions.
- Volatility Reduction: The "Volatility" metric (standard deviation of F1 scores across conditions) was significantly lower for the agentic approach, indicating better consistency. For TOEFL4, volatility was 2.5-4.1 for the agentic approach versus 12.4-36.5 for direct prompting.
- Ablation Study: The idiomatic expert was found to be the most informative single agent, but a combination of all experts yielded the best overall performance, suggesting complementarity.
Limitations and Future Work
- Scope of Adversarial Experiments: The current study focused on relatively salient, content-based hints (signatures, stereotypes). Future work should explore more sophisticated adversarial attacks that mimic linguistic features more subtly.
- Dataset Representativeness: The experiments used standard L2 learner corpora. Future work should evaluate the approach on more diverse, real-world scenarios, including low-resource settings with shorter texts.
- Cross-linguistic Generalizability: The study focused on English as L2. Further research is needed to assess the agentic approach's performance with other L1-L2 pairings.
- Optimizing Agent Interactions: Future work could refine agent interactions, develop more sophisticated evidence synthesis mechanisms for the coordinator, and explore dynamic weighting of agent contributions.
- External Knowledge Integration: Agents could be equipped with tools to consult external knowledge bases for more verifiable evidence.
- Ethical Considerations: While the study used anonymized data and is for research purposes, the authors acknowledge the broader societal implications of authorship analysis and the need for responsible deployment.
Summary updated