Using an LLM to Tackle Difficult Cases in NER for Nepali Tweets

Claire WangarXiv 2026 · arXiv

TL;DR

This paper presents a hybrid framework for Named Entity Recognition (NER) in Nepali tweets, a low-resource and noisy data environment. By using a fine-tuned Pretrained Language Model (PLM) to identify "hard" cases based on token probability and routing them to a Large Language Model (LLM) with optimized few-shot prompting, the authors achieved a performance boost over using the PLM alone.

What Problem Does This Paper Address?

Named Entity Recognition (NER)—the task of identifying and categorizing entities like people, places, and dates—is notoriously difficult for social media data. Tweets are noisy, lack context, and frequently feature "code-switching" (mixing multiple languages, such as Nepali and English).

For low-resource languages like Nepali, the challenge is amplified by a lack of labeled training data. While specialized, encoder-only Pretrained Language Models (PLMs) like DistilBERT currently hold the state-of-the-art for NER, they often struggle with these "hard" edge cases. Conversely, Large Language Models (LLMs) are powerful but expensive and generally underperform PLMs on standard sequence labeling tasks. The authors sought to determine if an LLM could act as a "specialist" for the difficult cases where the PLM is uncertain.

Methods

The authors developed a multi-stage pipeline to combine the strengths of both model types:

  1. Confidence-Based Routing: They fine-tuned a distilbert-base-nepali model on Nepali tweets. Any tweet in the test set containing a token with a prediction probability below 90% was classified as "Hard," while the rest were labeled "Easy."
  2. LLM Prompt Engineering: They evaluated an LLM (Gemma 4 26B-A4B) on the Hard set using 16 different prompt configurations. They varied:
  3. Number of examples: 0, 10, 20, or 40 few-shot examples.
  4. Selection strategy: Randomly chosen examples vs. Similarity-Based examples (chosen using cosine similarity of word embeddings).
  5. Definitions: Including or excluding explicit entity type definitions.
  6. Self-Verification: They tested a step where the LLM re-evaluated its own initial predictions to prune false positives.
  7. Hybrid Inference: The final system used the PLM for Easy cases and the best-performing LLM configuration for Hard cases.
Figure 1: Illustration of our framework. The pretrained language model sorts test tweets as easy or hard based on token probabilities; hard tweets are sent to the LLM for evaluation using each prompt configuration.

Datasets

The study utilized the DanfeNER dataset, a human-annotated collection of Nepali tweets. It includes five entity types: Person (PER), Location (LOC), Organization (ORG), Date (DATE), and Event (EVENT). The authors performed additional data cleaning to resolve formatting inconsistencies between different versions of the dataset.

Key Insights and Ideas

  • Semantic Similarity is King: The most significant performance gain came from few-shot example selection. Using examples semantically similar to the target tweet improved the F1 score by nearly 10 points compared to the baseline.
  • Definitions are Optional: Interestingly, providing explicit entity definitions did not meaningfully improve performance, contradicting some previous findings in English-centric NER research.
  • LLMs Excel at Code-Switching: The LLM showed a clear advantage in handling code-switched text (Nepali mixed with Latin script). It produced a higher percentage of true positives and fewer false negatives than the PLM in these instances.
  • Complementary Strengths: The LLM outperformed the PLM specifically on the "Hard" subset, justifying the routing strategy even though the PLM was superior on the "Easy" subset and the full dataset.

Key Results

The prompt engineering experiments (visualized in Figure 2) showed a clear upward trend in performance as the number of similar examples increased.

Figure 2: Graph of data in Table 2.

Quantitatively, the hybrid approach proved successful:

  • Standalone PLM F1: 80.78
  • Combined (PLM + LLM) F1: 81.76
  • On the Hard Test Set specifically, the LLM achieved an F1 of 64.97, surpassing the PLM's 63.23.

Detailed results for the final comparison can be seen in the table below:

Table 4: F1 scores showing the LLM outperforming the PLM on the Hard Test set and the Combined model achieving the highest overall score.

Limitations and Future Work

The authors noted several areas for improvement:

  • Small Hard Set: The "Hard" test set consisted of only 214 tweets, which may lead to high variance in results.
  • Modest Gains: The overall F1 increase was approximately 1%. To achieve more substantial gains, the authors suggest identifying a larger "hard" set by adjusting the probability threshold.
  • Prompting Refinement: While self-verification and definitions didn't help here, different implementations of these concepts might yield better results in future iterations.
  • Cost vs. Performance: LLM inference is significantly more expensive than PLM inference; future work could look at optimizing this trade-off.

Summary updated

Abstract

Named entity recognition (NER) is challenging for social media data in a low-resource language. LLMs often lag behind pretrained language models (PLMs) in NER performance, but they show promise in handling cases that PLMs struggle with. Using a dataset of Nepali tweets, we split the test set into "easy" and "hard" cases using token probabilities from a fine-tuned pretrained language model (PLM). We then evaluate an LLM on the hard test cases using 16 different prompt configurations. We find that selecting examples based on semantic similarity and increasing the number of examples improves F1 score by almost 10 points from the baseline configuration. Although the LLM underperforms the PLM on the full and easy test sets, it surpasses the PLM on the hard test set, resulting in an improved score when the two models are combined.
BibTeX
@misc{wang2026using,
  author = {Claire Wang},
  title = {{Using an LLM to Tackle Difficult Cases in NER for Nepali Tweets}},
  year = {2026},
  eprint = {2611.08967},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.08967},
}
Figure