Can LLMs Resolve Entity Conflicts in Domain Specific NLU Pipelines?

Akansha Bhardwaj, Mingbo Cui, Victor Guimaraes, Dominik Kohler, Diana Nicoleta Popa, Meghana M. Reddy, Paolo RossoarXiv 2026 · arXiv

TL;DR

This paper explores replacing brittle, hand-crafted rules in NLU pipelines with Large Language Models (LLMs) to resolve entity conflicts. By leveraging Llama 3.1-8B with a combination of LoRA fine-tuning and multi-shot prompting, the researchers achieved performance comparable to or better than traditional systems while significantly reducing the engineering effort required to maintain complex rule sets.

What Problem Does This Paper Address?

In modern Natural Language Understanding (NLU) pipelines, it is common to run multiple Named Entity Recognition (NER) extractors in parallel. While this modular approach is flexible, it frequently leads to entity conflicts: multiple extractors might claim the same span of text or provide overlapping predictions.

For example, in the phrase "oil change tire rotation," one extractor might identify "oil change" and "tire rotation" as separate repair operations, while another might incorrectly see "change tire" as a single overlapping entity. Similarly, the word "jimmy" could be a person's name or a GMC car model.

Fig. 1: An example of overlapping and conflicting entity spans from automotive domain. A rule-based extractor can incorrectly extract change tire as a repair operation from the utterance ‘oil change tire rotation’ where the caller intended repair operations are oil change and tire rotation. To resolve this conflict, a rule-based conflict resolution strategy leverages the span information. Similarly, a data driven NER incorrectly extracts jimmy as a name entity type, but jimmy is a car model from GMC, a division of American automotive manufacturer General Motors. To resolve this conflict, a rule-based conflict resolution strategy leverages the intent of the utterance and the presence of other entities. Such rule-based conflict resolution strategies are specific to each use-case and tedious to maintain.

Traditionally, developers resolve these using sequential priority (the first extractor wins) or complex, hand-coded rules. However, these rules are "tedious to maintain," require deep domain expertise, and often break when new services or entities are added to the system.

Methods

The authors propose a single LLM-based component to replace the entire stack of resolution rules. They tested several strategies using the Llama 3.1-8B-Instruct model:

  1. Multi-shot Prompting: Providing the model with 1 to 8 manually curated examples of correct resolutions.
  2. Example-Based Retrieval (EBR): Instead of static examples, the system uses cosine similarity to retrieve the most relevant examples from a validation set for every query.
  3. LoRA Fine-tuning: Using Low-Rank Adaptation to fine-tune the model on domain-specific data, making it more computationally efficient than full fine-tuning while capturing niche patterns.

The LLM acts as a "black-box" arbitrator: it receives the original text, the conversation context (e.g., "service request"), and the list of candidate entities, then outputs the final, resolved set.

Datasets

The study utilized two primary datasets focused on the automotive domain:

  • Automotive-Rule-Based: A large-scale dataset of over 60,000 utterances from a real-world dialogue system. Labels were generated by the existing rule-based pipeline to see if an LLM could "learn" the current system's behavior.
  • Automotive-Curated: A high-quality "gold standard" benchmark consisting of 674 ambiguous examples. Each example was human-annotated by two experts who agreed on the correct resolution.

The data was categorized into three contexts: Car Details, Service Request, and Time Negotiation.

Key Insights and Ideas

  • General vs. Domain Knowledge: The base LLM performed exceptionally well on "Time Negotiation" (a general task) but struggled with "Service Request" (a domain-specific task). This confirms that while LLMs are "smart," they still require domain-specific grounding to handle niche business logic.
  • The Power of Fine-tuning: For domain-specific cases, LoRA fine-tuning provided the biggest performance leaps. In the "Service Request" context, fine-tuning increased accuracy by nearly 60% compared to the base model.
  • The "Recalibration" Dip: Interestingly, the researchers noticed that adding a new example to a prompt sometimes caused a temporary performance drop (as seen in the fluctuations in Figure 2/5). They hypothesize the model needs to "recalibrate" to the diversity introduced by new patterns before its performance stabilizes again.
  • Retrieval Wins for Consistency: Using Example-Based Retrieval (EBR) led to a much steadier upward trend in improvement compared to manually curated examples, which were more sensitive to the quality of the specific examples chosen.
Fig. 3: Comparison of trends in improvement when examples in multi-shot prompting are manually curated versus when they are obtained from Example-Based Retrieval pipeline. Results shown above are obtained using Llama 3.1-8B-Instruct model on Automotive-Rule Based dataset. a) car details context b) time negotiation context c) service request context.

Key Results

  • Competitive Accuracy: On the human-validated Automotive-Curated dataset, the best LLM approach (LoRA + 7-shot) achieved 78.93% accuracy, slightly outperforming the traditional rule-based system (78.60%).
  • Superior Generalization: The LLM outperformed the rule-based system in the "Time Negotiation" context by 8.27%, proving it is much better at handling the nuances of natural language than rigid rules.
  • High Agreement: The fine-tuned model achieved an 81.42% agreement rate with the existing rule-based system, suggesting it is a viable drop-in replacement.
  • Base Model Limitations: Without fine-tuning or examples (0-shot), the base Llama model performed poorly (under 9% matching), highlighting that "off-the-shelf" LLMs are not ready for domain-specific NLU tasks without intervention.

Limitations and Future Work

The authors acknowledge that latency is a potential trade-off; calling an LLM is computationally more expensive than executing a set of if-else statements. Additionally, while the LLM excelled at general contexts, there is still a slight performance gap in highly specific domain contexts like "Service Request" where the rule-based system still holds a narrow lead.

Future work will focus on more sophisticated retrieval strategies that prioritize diversity in the examples provided to the model, potentially smoothing out the performance fluctuations seen during multi-shot prompting.

Summary updated

Abstract

Resolving entity conflicts is a critical step in Pipeline-based NLU architectures, especially when multiple extractors provide overlapping or conflicting outputs. Traditional rule-based entity conflict resolution methods often lack flexibility and are unscalable. In this work, we aim to study if the broad knowledge of LLMs can be leveraged to resolve entity conflicts in domain specific NLU pipelines. We present an experimental comparison between a traditional rule-based conflict resolution method, and two LLM-based approaches: a multi-shot prompting method using the Llama3.1-8B-Instruct model and a LoRA fine-tuned Llama3.1-8B-Instruct model. The evaluation was conducted under two setups: (1) a broad assessment of LLM's capability to match existing NLU pipeline output, and (2) a more controlled scenario assessing the LLM accuracy on human-labeled data. Results show that while multi-shot prompting offers ease of adaptation and rapid behavioral changes, the fine-tuned model delivers superior performance on complex, ambiguous cases. Ultimately, the best approach appears to be a combination of both strategies, leveraging multi-shot prompting's flexibility alongside the broad accuracy gains of LoRA fine-tuning for a more comprehensive performance.
BibTeX
@misc{bhardwaj2026can,
  author = {Akansha Bhardwaj and Mingbo Cui and Victor Guimaraes and Dominik Kohler and Diana Nicoleta Popa and Meghana M. Reddy and Paolo Rosso},
  title = {{Can LLMs Resolve Entity Conflicts in Domain Specific NLU Pipelines?}},
  year = {2026},
  eprint = {2611.25733},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25733},
}
Figure