Information Extraction from Conversation Transcripts: Neuro-Symbolic vs. LLM

Alice Saebom Kwak, Maria Alexeeva, Gus Hahn-Powell, Keith Alcock, Kevin McLaughlin, Doug McCorkle, Gabe McNunn, Mihai SurdeanuarXiv 2025 · DOI · arXiv

Abstract

Extracting structured information from natural conversation is a complex task due to speech recognition errors, disfluencies, and non-linear dialogue. This paper compares a traditional neuro-symbolic (NS) system using rule-based extraction with a modern Large Language Model (LLM) approach. Using a specialized agricultural dataset, the study finds that while LLMs offer superior generalization and higher F1 scores (69.4 vs. 52.7), neuro-symbolic systems provide significant advantages in processing speed and deterministic control.

Introduction

The research community has increasingly shifted toward Large Language Models (LLMs) for information extraction (IE) tasks, often overlooking the benefits of symbolic AI. This study investigates the "hidden costs" of this shift by evaluating both approaches on "noisy" interview transcripts. These transcripts are characterized by ASR errors (e.g., "Souths" instead of "sows") and information split across multiple speaker turns, making structured data extraction particularly challenging.

Methodology: Two Distinct Pipelines

The authors developed two parallel pipelines to process agricultural interviews. Both share initial preprocessing but diverge in their core extraction logic.

Comparison of the Neuro-Symbolic and LLM-based extraction architectures.
  1. The Neuro-Symbolic (NS) System: This pipeline utilizes a neural backbone for low-level NLP tasks and the Odin framework for rule-based extraction. It includes a dedicated Dialogue Management component to link interviewer questions to farmer responses across turns.
  2. The LLM-based System: Utilizing a quantized OpenChat 3.5 model, this system employs keyword-based topic segmentation to manage long transcripts. It outputs structured JSON via Pydantic dataclasses and includes a verification layer to filter hallucinations by grounding outputs in the source text.

Performance Evaluation

The systems were tested on a new benchmark consisting of nine information-dense agricultural interviews covering pork, dairy, and crops. The results highlight a significant performance gap in favor of the LLM-based approach.

Performance metrics (Precision, Recall, and F1) across Pork, Crop, and Dairy domains.

The LLM-based system achieved a total F1 score of 69.4, compared to 52.7 for the NS system. This difference is primarily driven by a substantial "Recall Gap": the LLM (67.4 recall) was far more effective at identifying information phrased in unexpected ways than the rigid rule-based NS system (44.6 recall). However, the precision gap was narrower (72.5 vs. 66.6), indicating that the NS system is relatively reliable when it does trigger a rule.

The Efficiency Paradox and Technical Debt

Despite the LLM's accuracy, the NS system demonstrated a massive advantage in computational efficiency. On a standard CPU, the NS system processed the dataset in 69 seconds, whereas the LLM required over 6 hours. Even with GPU acceleration, the LLM remained slower than the NS system running on a basic laptop processor.

Furthermore, the authors highlight "technical debt" associated with LLMs. While LLMs are faster to deploy initially, they function as "black boxes," making them harder to debug than the transparent, rule-based NS system. If the NS system fails, a developer can pinpoint exactly which rule was missing or incorrect.

Conclusion

The study concludes that while LLMs are currently superior for handling the linguistic variety of natural conversation, neuro-symbolic systems remain highly relevant for resource-constrained or high-precision industrial environments. The authors propose that future industrial IE solutions should likely be hybrid systems, combining the generalization of LLMs with the controllability and efficiency of symbolic logic.

Summary updated

Abstract

The current trend in information extraction (IE) is to rely extensively on large language models, effectively discarding decades of experience in building symbolic or statistical IE systems. This paper compares a neuro-symbolic (NS) and an LLM-based IE system in the agricultural domain, evaluating them on nine interviews across pork, dairy, and crop subdomains. The LLM-based system outperforms the NS one (F1 total: 69.4 vs. 52.7; core: 63.0 vs. 47.2), where total includes all extracted information and core focuses on essential details. However, each system has trade-offs: the NS approach offers faster runtime, greater control, and high accuracy in context-free tasks but lacks generalizability, struggles with contextual nuances, and requires significant resources to develop and maintain. The LLM-based system achieves higher performance, faster deployment, and easier maintenance but has slower runtime, limited control, model dependency and hallucination risks. Our findings highlight the "hidden cost" of deploying NLP systems in real-world applications, emphasizing the need to balance performance, efficiency, and control.
BibTeX
@misc{kwak2025information,
  author = {Alice Saebom Kwak and Maria Alexeeva and Gus Hahn-Powell and Keith Alcock and Kevin McLaughlin and Doug McCorkle and Gabe McNunn and Mihai Surdeanu},
  title = {{Information Extraction from Conversation Transcripts: Neuro-Symbolic vs. LLM}},
  year = {2025},
  eprint = {2510.12023},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.12023},
}
Figure