Abstract
In this paper, we assess the robustness (reliability) of ChatGPT under input perturbations for one of the most fundamental tasks of Information Extraction (IE) i.e. Named Entity Recognition (NER). Despite the hype, the majority of the researchers have vouched for its language understanding and generation capabilities; a little attention has been paid to understand its robustness: How the input-perturbations affect 1) the predictions, 2) the confidence of predictions and 3) the quality of rationale behind its prediction. We perform a systematic analysis of ChatGPT's robustness (under both zero-shot and few-shot setup) on two NER datasets using both automatic and human evaluation. Based on automatic evaluation metrics, we find that 1) ChatGPT is more brittle on Drug or Disease replacements (rare entities) compared to the perturbations on widely known Person or Location entities, 2) the quality of explanations for the same entity considerably differ under different types of "Entity-Specific" and "Context-Specific" perturbations and the quality can be significantly improved using in-context learning, and 3) it is overconfident for majority of the incorrect predictions, and hence it could lead to misguidance of the end-users.
Introduction
Large Language Models (LLMs) like ChatGPT have revolutionized natural language processing, yet their reliability in technical domains like Information Extraction (IE) remains a critical concern. While these models excel at general conversation, tasks like Named Entity Recognition (NER) require high precision and robustness. This study investigates the "robustness gap"—the discrepancy between a model's performance on clean data versus the messy, perturbed data encountered in real-world scenarios, such as text containing typos, synonyms, or varying contexts.
Methodology
The researchers evaluated the robustness of ChatGPT (specifically the gpt-3.5-turbo model) using a comprehensive framework of automated and manual perturbations. These perturbations were designed to test the model's reliance on memorized knowledge versus linguistic context.
Perturbation Types
The study employed two primary categories of perturbations:
- Entity-Specific Perturbations: These involve modifying the target entity itself. Examples include replacing an entity with an alias (synonym), a different entity of the same type, a version with typos, or a completely random string.
- Context-Specific Perturbations: These involve changing the surrounding sentence structure, such as replacing verbs with synonyms using BERT-based suggestions, to see if the model's understanding of the entity's role is affected.
Datasets
The evaluation spanned two distinct NER benchmarks:
- BC5CDR: A specialized biomedical dataset focusing on Chemicals and Diseases.
- CoNLL-2003: A general-domain dataset focusing on Person and Location entities.
This dual-dataset approach allowed researchers to compare ChatGPT's performance on widely known "common knowledge" entities versus specialized, domain-specific "rare" entities.
Key Insights and Results
Domain-Specific Brittleness
A major finding is that ChatGPT is significantly more fragile when dealing with specialized biomedical entities compared to general-domain entities. The model appears to rely more on memorized "world knowledge" for drugs and diseases, making it highly susceptible to failure when those specific strings are altered. In zero-shot tests on the BC5CDR dataset, "Random Entity" perturbations caused the largest accuracy drop ($\Delta$ Accuracy of 0.38), followed by typos (0.30).
The Overconfidence Trap
In zero-shot settings, ChatGPT frequently exhibits overconfidence in its errors. When a perturbation leads to an incorrect prediction, the model often maintains a high confidence score (e.g., 0.8 or 0.9). This lack of calibration is a significant risk for end-users who might be misled by a convincingly stated but incorrect extraction.
Rationale Shifts: Global vs. Local
The study analyzed the rationales provided by ChatGPT for its predictions. When an entity is clear and unperturbed, the model typically provides "Global" explanations (e.g., "This is a known medication"). However, when faced with perturbations, a successful model must shift to "Local" explanations based on linguistic context (e.g., "Based on the context of 'infusion,' this must be a chemical").
The Impact of Few-Shot Learning
Transitioning from zero-shot to few-shot In-Context Learning (ICL) yielded dramatic improvements. For instance, providing a few examples reduced the accuracy drop for aliases in the biomedical dataset from 0.16 to just 0.03. Furthermore, few-shot learning improved the "faithfulness" of rationales and helped calibrate the model's confidence, increasing the gap between confidence scores for correct versus incorrect predictions.
Conclusion
While ChatGPT is a powerful tool for NER, its sensitivity to input perturbations highlights a need for caution in sensitive domains. The model's tendency toward overconfidence in zero-shot scenarios can be mitigated through few-shot learning, which enhances both accuracy and reliability. Future research should focus on developing prompting strategies that better balance local context and global knowledge to ensure robust performance in out-of-distribution scenarios.
Summary updated