Abstract
Extracting structured information from natural conversation is a complex task due to speech recognition errors, disfluencies, and non-linear dialogue. This paper compares a traditional neuro-symbolic (NS) system using rule-based extraction with a modern Large Language Model (LLM) approach. Using a specialized agricultural dataset, the study finds that while LLMs offer superior generalization and higher F1 scores (69.4 vs. 52.7), neuro-symbolic systems provide significant advantages in processing speed and deterministic control.
Introduction
The research community has increasingly shifted toward Large Language Models (LLMs) for information extraction (IE) tasks, often overlooking the benefits of symbolic AI. This study investigates the "hidden costs" of this shift by evaluating both approaches on "noisy" interview transcripts. These transcripts are characterized by ASR errors (e.g., "Souths" instead of "sows") and information split across multiple speaker turns, making structured data extraction particularly challenging.
Methodology: Two Distinct Pipelines
The authors developed two parallel pipelines to process agricultural interviews. Both share initial preprocessing but diverge in their core extraction logic.
- The Neuro-Symbolic (NS) System: This pipeline utilizes a neural backbone for low-level NLP tasks and the Odin framework for rule-based extraction. It includes a dedicated Dialogue Management component to link interviewer questions to farmer responses across turns.
- The LLM-based System: Utilizing a quantized OpenChat 3.5 model, this system employs keyword-based topic segmentation to manage long transcripts. It outputs structured JSON via Pydantic dataclasses and includes a verification layer to filter hallucinations by grounding outputs in the source text.
Performance Evaluation
The systems were tested on a new benchmark consisting of nine information-dense agricultural interviews covering pork, dairy, and crops. The results highlight a significant performance gap in favor of the LLM-based approach.
The LLM-based system achieved a total F1 score of 69.4, compared to 52.7 for the NS system. This difference is primarily driven by a substantial "Recall Gap": the LLM (67.4 recall) was far more effective at identifying information phrased in unexpected ways than the rigid rule-based NS system (44.6 recall). However, the precision gap was narrower (72.5 vs. 66.6), indicating that the NS system is relatively reliable when it does trigger a rule.
The Efficiency Paradox and Technical Debt
Despite the LLM's accuracy, the NS system demonstrated a massive advantage in computational efficiency. On a standard CPU, the NS system processed the dataset in 69 seconds, whereas the LLM required over 6 hours. Even with GPU acceleration, the LLM remained slower than the NS system running on a basic laptop processor.
Furthermore, the authors highlight "technical debt" associated with LLMs. While LLMs are faster to deploy initially, they function as "black boxes," making them harder to debug than the transparent, rule-based NS system. If the NS system fails, a developer can pinpoint exactly which rule was missing or incorrect.
Conclusion
The study concludes that while LLMs are currently superior for handling the linguistic variety of natural conversation, neuro-symbolic systems remain highly relevant for resource-constrained or high-precision industrial environments. The authors propose that future industrial IE solutions should likely be hybrid systems, combining the generalization of LLMs with the controllability and efficiency of symbolic logic.
Summary updated