TL;DR
This paper presents a hybrid framework for Named Entity Recognition (NER) in Nepali tweets, a low-resource and noisy data environment. By using a fine-tuned Pretrained Language Model (PLM) to identify "hard" cases based on token probability and routing them to a Large Language Model (LLM) with optimized few-shot prompting, the authors achieved a performance boost over using the PLM alone.
What Problem Does This Paper Address?
Named Entity Recognition (NER)—the task of identifying and categorizing entities like people, places, and dates—is notoriously difficult for social media data. Tweets are noisy, lack context, and frequently feature "code-switching" (mixing multiple languages, such as Nepali and English).
For low-resource languages like Nepali, the challenge is amplified by a lack of labeled training data. While specialized, encoder-only Pretrained Language Models (PLMs) like DistilBERT currently hold the state-of-the-art for NER, they often struggle with these "hard" edge cases. Conversely, Large Language Models (LLMs) are powerful but expensive and generally underperform PLMs on standard sequence labeling tasks. The authors sought to determine if an LLM could act as a "specialist" for the difficult cases where the PLM is uncertain.
Methods
The authors developed a multi-stage pipeline to combine the strengths of both model types:
- Confidence-Based Routing: They fine-tuned a
distilbert-base-nepalimodel on Nepali tweets. Any tweet in the test set containing a token with a prediction probability below 90% was classified as "Hard," while the rest were labeled "Easy." - LLM Prompt Engineering: They evaluated an LLM (
Gemma 4 26B-A4B) on the Hard set using 16 different prompt configurations. They varied: - Number of examples: 0, 10, 20, or 40 few-shot examples.
- Selection strategy: Randomly chosen examples vs. Similarity-Based examples (chosen using cosine similarity of word embeddings).
- Definitions: Including or excluding explicit entity type definitions.
- Self-Verification: They tested a step where the LLM re-evaluated its own initial predictions to prune false positives.
- Hybrid Inference: The final system used the PLM for Easy cases and the best-performing LLM configuration for Hard cases.
Datasets
The study utilized the DanfeNER dataset, a human-annotated collection of Nepali tweets. It includes five entity types: Person (PER), Location (LOC), Organization (ORG), Date (DATE), and Event (EVENT). The authors performed additional data cleaning to resolve formatting inconsistencies between different versions of the dataset.
Key Insights and Ideas
- Semantic Similarity is King: The most significant performance gain came from few-shot example selection. Using examples semantically similar to the target tweet improved the F1 score by nearly 10 points compared to the baseline.
- Definitions are Optional: Interestingly, providing explicit entity definitions did not meaningfully improve performance, contradicting some previous findings in English-centric NER research.
- LLMs Excel at Code-Switching: The LLM showed a clear advantage in handling code-switched text (Nepali mixed with Latin script). It produced a higher percentage of true positives and fewer false negatives than the PLM in these instances.
- Complementary Strengths: The LLM outperformed the PLM specifically on the "Hard" subset, justifying the routing strategy even though the PLM was superior on the "Easy" subset and the full dataset.
Key Results
The prompt engineering experiments (visualized in Figure 2) showed a clear upward trend in performance as the number of similar examples increased.
Quantitatively, the hybrid approach proved successful:
- Standalone PLM F1: 80.78
- Combined (PLM + LLM) F1: 81.76
- On the Hard Test Set specifically, the LLM achieved an F1 of 64.97, surpassing the PLM's 63.23.
Detailed results for the final comparison can be seen in the table below:
Limitations and Future Work
The authors noted several areas for improvement:
- Small Hard Set: The "Hard" test set consisted of only 214 tweets, which may lead to high variance in results.
- Modest Gains: The overall F1 increase was approximately 1%. To achieve more substantial gains, the authors suggest identifying a larger "hard" set by adjusting the probability threshold.
- Prompting Refinement: While self-verification and definitions didn't help here, different implementations of these concepts might yield better results in future iterations.
- Cost vs. Performance: LLM inference is significantly more expensive than PLM inference; future work could look at optimizing this trade-off.
Summary updated