Abstract
Social robots require the ability to recognize fine-grained, domain-specific entities to engage in meaningful human interaction. However, existing Named Entity Recognition (NER) datasets are often too generic or too small for specialized applications. We introduce RapidNER, a framework that accelerates dataset construction by combining Knowledge Graph (KG) triples with automated Elasticsearch-based tagging. Using this method, we produced NERsocial, a large-scale dataset featuring 99.4K sentences across six entity types (Drink, Food, Hobby, Job, Pet, Sport). Our results demonstrate that RapidNER reduces annotation time to milliseconds per sentence while producing models with high in-domain performance (F1 > 96%) and superior generalization capabilities.
Introduction
As social robots transition from research labs to real-world environments like homes and restaurants, their natural language understanding capabilities must evolve. A robot serving a customer needs to distinguish between specific food items ("Taco salad") or understand a user's profession ("archivist") to maintain contextually relevant dialogue.
Standard NER systems trained on broad datasets like CoNLL-2003 fail to capture these nuances. The primary bottleneck is the "data hunger" of modern transformer models; manually labeling the thousands of sentences required for niche domains is prohibitively expensive. We address this by proposing RapidNER, a framework for rapid, high-quality data acquisition and annotation.
Methodology: The RapidNER Framework
The RapidNER framework operates through a three-phase pipeline designed to minimize human effort while maximizing data diversity and accuracy.
1. Knowledge Acquisition
We extract domain-specific sub-graphs from Wikidata by traversing instance-of and subclass-of relations. This process generates massive dictionaries of "hyponyms"—specific examples for target categories such as Drink, Food, Hobby, Job, Pet, and Sport.
2. Multi-Source Text Harvesting
To ensure the model is robust to different registers of speech, we pull text from three distinct sources:
- Wikipedia: For formal, descriptive narratives.
- Reddit: For conversational parlance from specialized subreddits (e.g., r/Hobbies).
- Stack Exchange: For technical but informal forum discussions (e.g., Cooking Stack Exchange).
3. Automated Annotation via Elasticsearch
The core innovation of RapidNER is the use of Elasticsearch (ES) for automated tagging. By importing harvested sentences into ES and utilizing its Fast Vector Highlighter (fvh), we treat the KG-derived dictionary as a search space. This allows the system to demarcate entity spans across thousands of sentences in approximately 0.9 milliseconds per sentence. A final human verification pass is conducted to correct any edge cases or missed tags.
The NERsocial Dataset
The resulting dataset, NERsocial, serves as a new benchmark for HRI-focused NER. It comprises 153,102 tokens and 99,448 sentences. Compared to common baselines like WNUT2017, which contains only ~5.6K sentences, NERsocial provides a much higher density of specialized entities, making it ideal for training high-capacity transformer models.
Results and Discussion
Fine-tuned models including BERT, RoBERTa, and DeBERTa-v3 achieved average F1-scores between 95.98% and 96.41% on NERsocial.
Cross-Domain Generalization
A key finding of our study is the "Wikipedia Trap." Models trained solely on Wikipedia text experienced a 50% drop in F1-score when tested on informal social media data. This highlights the necessity of multi-source training. By incorporating Reddit and Stack Exchange, NERsocial-trained models achieved significantly better out-of-domain (OOD) generalization. For instance, a DeBERTa model trained on NERsocial achieved a 13.57% F1 on the WNUT dataset, whereas the reverse transfer (WNUT to NERsocial) resulted in only 1.46% F1.
Efficiency Gains
The transition from manual annotation (approx. 1 minute/sentence) to the RapidNER ES-based scheme (0.9 ms/sentence) represents a transformative speedup for dataset curation. This efficiency allows for the rapid adaptation of NER systems to new emerging domains or cultural shifts.
Conclusion
RapidNER provides a scalable solution to the data scarcity problem in specialized NER. By leveraging structured knowledge from Wikidata and the search efficiency of Elasticsearch, we have created NERsocial, a robust dataset for social robotics. Future work will explore the integration of Entity Linking to connect recognized mentions to unique knowledge base identifiers and address potential cultural biases in the source platforms.
Summary updated