ON THE ROBUSTNESS OF CHATGPT UNDER INPUT PERTURBATIONS FOR NAMED ENTITY RECOGNITION TASK

Ishani Mondal, Abhilasha SanchetiICLR 2024 · OpenReview

TL;DR

This paper investigates how easily ChatGPT’s Named Entity Recognition (NER) capabilities can be "broken" by small changes to input text, such as typos, synonyms, or swapping entities. Researchers found that while ChatGPT is fairly robust with famous people and places, it struggles significantly with specialized domains like medicine. However, providing just a few examples (few-shot learning) drastically improves its resilience and the quality of its explanations.

The Challenge of Real-World NER

Named Entity Recognition (NER)—the task of identifying and categorizing entities like "London" (Location) or "Aspirin" (Drug)—is a cornerstone of natural language processing. While ChatGPT has shown impressive zero-shot capabilities, its reliability in the "wild" remains a concern. In real-world applications, text is messy: it contains typos, rare technical terms, and varied phrasing.

The authors address a critical gap: How robust is ChatGPT when the input deviates from its training distribution? If a model fails to recognize a life-saving drug just because of a typo or because it was replaced by a similar-sounding chemical, its utility in high-stakes fields like medicine or law is severely compromised.

Experimental Setup: Perturbing the Input

The researchers evaluated ChatGPT (specifically gpt-3.5-turbo) in two scenarios: zero-shot (no examples provided) and few-shot (four examples per entity type). They applied two categories of intentional perturbations to the input text across two standard benchmarks: CoNLL-2003 (general NER: Person/Location) and BC5CDR (biomedical NER: Chemical/Disease).

Examples of original sentences versus their perturbed counterparts

The perturbations included:

  1. Entity-specific Perturbations:
    • Alias: Replacing an entity with a different name for the same thing (via Wikidata).
  2. Same Entity Type: Swapping one entity (e.g., "Brain damage") with another of the same category ("Chorioretinal atrophy").
  3. Typo: Rotating characters to create natural-looking spelling errors.
  4. Random String: Replacing the entity with a nonsense string to see if the model can still identify it based purely on context.
  5. Context-specific Perturbations:
    • Verb Substitution: Using BERT to find and replace the main verb in a sentence with a synonym to see if changing the "action" confuses the model about the "subject."

Beyond simple accuracy, the evaluation measured the faithfulness and globalness of the model's explanations—checking if the model's reasoning was grounded in the immediate sentence (local) or its internal "world knowledge" (global).

Key Findings: Brittleness and the Power of Few-Shot Learning

The results reveal a stark contrast in how ChatGPT handles different domains and learning settings.

Performance drops across different perturbations in zero-shot vs. few-shot settings
  • Domain Matters: ChatGPT is much more "brittle" when dealing with rare or specialized entities (Drugs/Diseases) than with common ones (People/Places). It likely encountered "Paris" millions of times during training, but "fucoidan" significantly fewer times, making its recognition of the latter highly dependent on perfect input.
  • Accuracy Drops in Zero-Shot: In the zero-shot setting, Random Entity and Typo perturbations caused the largest performance hits. For the biomedical dataset (BC5CDR), accuracy dropped by as much as 0.38 under random perturbations.
  • Few-Shot is a Force Multiplier: The jump in robustness from zero-shot to few-shot is dramatic. By seeing just four examples, the model learns the "pattern" of the task, helping it ignore the noise of typos or random strings. The change in accuracy ($\Delta$ Accuracy) decreased significantly across almost all categories in the few-shot setting.
  • The "Survival Mechanism" of Context: When an entity is perturbed (e.g., via a typo), the model's reasoning shifts. It moves away from relying on its internal "encyclopedia" (global knowledge) and leans more heavily on the surrounding words (local context) to guess what the entity is.
  • Explanation Quality is Volatile: The quality of ChatGPT's explanations varies wildly depending on how the input is tweaked. A typo might not just cause a wrong prediction; it can lead the model to provide a nonsensical or unfaithful explanation for its choice.

Limitations and Future Directions

The authors acknowledge several constraints in their study. For the CoNLL dataset, they limited testing to Person and Location entities to simplify the perturbation process. Additionally, the study focused on gpt-3.5-turbo; newer models like GPT-4 might exhibit different robustness profiles. Finally, measuring the "quality" of an explanation automatically is inherently difficult. The authors used cosine similarity and Wikipedia summaries as proxies, which may not capture the full nuance of human-like reasoning. Future work should explore more complex context-specific attacks beyond simple verb substitutions to further stress-test LLM robustness.

Summary updated

Abstract

We present a systematic evaluation of the robustness of ChatGPT (in both zeroand few-shot settings) under input perturbations for Named Entity Recognition (NER) task. Our findings suggest: (1) ChatGPT is more brittle on Drug or Disease entity perturbations (rare entities) as compared to those on widely known Person or Location entities, and (2) the quality of explanations (localness and globalness) for the same entity considerably differ under various Entity-specific and Contextspecific perturbations; the quality significantly improves using in-context learning.
BibTeX
@inproceedings{mondal2024robustness,
  author = {Ishani Mondal and Abhilasha Sancheti},
  title = {{ON THE ROBUSTNESS OF CHATGPT UNDER INPUT PERTURBATIONS FOR NAMED ENTITY RECOGNITION TASK}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=cyN5Ck1RFT},
}
Figure