NERsocial: Efficient Named Entity Recognition Dataset Construction for Human-Robot Interaction Utilizing RapidNER

Jesse Atuhurra, Hidetaka Kamigaito, Hiroki Ouchi, Hiroyuki Shindo, Taro WatanabearXiv 2024 · DOI · arXiv

Abstract

Social robots require the ability to recognize fine-grained, domain-specific entities to engage in meaningful human interaction. However, existing Named Entity Recognition (NER) datasets are often too generic or too small for specialized applications. We introduce RapidNER, a framework that accelerates dataset construction by combining Knowledge Graph (KG) triples with automated Elasticsearch-based tagging. Using this method, we produced NERsocial, a large-scale dataset featuring 99.4K sentences across six entity types (Drink, Food, Hobby, Job, Pet, Sport). Our results demonstrate that RapidNER reduces annotation time to milliseconds per sentence while producing models with high in-domain performance (F1 > 96%) and superior generalization capabilities.

Introduction

As social robots transition from research labs to real-world environments like homes and restaurants, their natural language understanding capabilities must evolve. A robot serving a customer needs to distinguish between specific food items ("Taco salad") or understand a user's profession ("archivist") to maintain contextually relevant dialogue.

Social robots require domain-specific NER to understand user preferences and identities during interaction.

Standard NER systems trained on broad datasets like CoNLL-2003 fail to capture these nuances. The primary bottleneck is the "data hunger" of modern transformer models; manually labeling the thousands of sentences required for niche domains is prohibitively expensive. We address this by proposing RapidNER, a framework for rapid, high-quality data acquisition and annotation.

Methodology: The RapidNER Framework

The RapidNER framework operates through a three-phase pipeline designed to minimize human effort while maximizing data diversity and accuracy.

The RapidNER pipeline: from Wikidata knowledge acquisition to automated annotation and model fine-tuning.

1. Knowledge Acquisition

We extract domain-specific sub-graphs from Wikidata by traversing instance-of and subclass-of relations. This process generates massive dictionaries of "hyponyms"—specific examples for target categories such as Drink, Food, Hobby, Job, Pet, and Sport.

2. Multi-Source Text Harvesting

To ensure the model is robust to different registers of speech, we pull text from three distinct sources:

  • Wikipedia: For formal, descriptive narratives.
  • Reddit: For conversational parlance from specialized subreddits (e.g., r/Hobbies).
  • Stack Exchange: For technical but informal forum discussions (e.g., Cooking Stack Exchange).

3. Automated Annotation via Elasticsearch

The core innovation of RapidNER is the use of Elasticsearch (ES) for automated tagging. By importing harvested sentences into ES and utilizing its Fast Vector Highlighter (fvh), we treat the KG-derived dictionary as a search space. This allows the system to demarcate entity spans across thousands of sentences in approximately 0.9 milliseconds per sentence. A final human verification pass is conducted to correct any edge cases or missed tags.

The NERsocial Dataset

The resulting dataset, NERsocial, serves as a new benchmark for HRI-focused NER. It comprises 153,102 tokens and 99,448 sentences. Compared to common baselines like WNUT2017, which contains only ~5.6K sentences, NERsocial provides a much higher density of specialized entities, making it ideal for training high-capacity transformer models.

Results and Discussion

Fine-tuned models including BERT, RoBERTa, and DeBERTa-v3 achieved average F1-scores between 95.98% and 96.41% on NERsocial.

Cross-Domain Generalization

A key finding of our study is the "Wikipedia Trap." Models trained solely on Wikipedia text experienced a 50% drop in F1-score when tested on informal social media data. This highlights the necessity of multi-source training. By incorporating Reddit and Stack Exchange, NERsocial-trained models achieved significantly better out-of-domain (OOD) generalization. For instance, a DeBERTa model trained on NERsocial achieved a 13.57% F1 on the WNUT dataset, whereas the reverse transfer (WNUT to NERsocial) resulted in only 1.46% F1.

Efficiency Gains

The transition from manual annotation (approx. 1 minute/sentence) to the RapidNER ES-based scheme (0.9 ms/sentence) represents a transformative speedup for dataset curation. This efficiency allows for the rapid adaptation of NER systems to new emerging domains or cultural shifts.

Conclusion

RapidNER provides a scalable solution to the data scarcity problem in specialized NER. By leveraging structured knowledge from Wikidata and the search efficiency of Elasticsearch, we have created NERsocial, a robust dataset for social robotics. Future work will explore the integration of Entity Linking to connect recognized mentions to unique knowledge base identifiers and address potential cultural biases in the source platforms.

Summary updated

Abstract

Adapting named entity recognition (NER) methods to new domains poses significant challenges. We introduce RapidNER, a framework designed for the rapid deployment of NER systems through efficient dataset construction. RapidNER operates through three key steps: (1) extracting domain-specific sub-graphs and triples from a general knowledge graph, (2) collecting and leveraging texts from various sources to build the NERsocial dataset, which focuses on entities typical in human-robot interaction, and (3) implementing an annotation scheme using Elasticsearch (ES) to enhance efficiency. NERsocial, validated by human annotators, includes six entity types, 153K tokens, and 99.4K sentences, demonstrating RapidNER's capability to expedite dataset creation.
BibTeX
@misc{atuhurra2024nersocial,
  author = {Jesse Atuhurra and Hidetaka Kamigaito and Hiroki Ouchi and Hiroyuki Shindo and Taro Watanabe},
  title = {{NERsocial: Efficient Named Entity Recognition Dataset Construction for Human-Robot Interaction Utilizing RapidNER}},
  year = {2024},
  eprint = {2412.09634},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.09634},
}
Figure