TL;DR
This paper introduces DepChecker, a fast, unsupervised, and explainable fact-checking system specifically designed for machine-generated text in conversational e-commerce settings. It leverages dependency parsing to extract claims from text and verify them against structured product catalogs, outperforming general-purpose fact-checking models in this specialized domain.
What Problem Does This Paper Address?
Natural Language Generation (NLG) systems are increasingly used to create dynamic, conversational responses in various applications, including e-commerce. While these systems offer flexibility over templates, they can sometimes generate factually incorrect or nonsensical information (hallucinations). This is particularly problematic in high-stakes domains like health or finance, but also in e-commerce where inaccurate product descriptions can lead to poor customer decisions and damage business reputation. Existing fact-checking methods, often developed for general-purpose structured data or unstructured text, struggle to perform well on the specific characteristics of NLG-generated text in conversational e-commerce, which often involves structured product data and a relatively consistent linguistic style. The authors aim to bridge this gap by developing a specialized, efficient, and explainable fact-checking system for this domain.
Methods
The core of DepChecker lies in its rule-based approach that utilizes dependency parsing. The system has three main components:
- Dependency Parsing and Rule Application: It parses the input text to identify product and attribute mentions. Rules are developed based on dependency relations (e.g.,
nsubj,attr,conj) to pinpoint the head tokens of these entities. - Entity Span Completion: After identifying head tokens, additional rules (using regular expressions, part-of-speech tags, etc.) are applied to extract the complete entity mentions (product names, attribute names, and their values).
- Fact Verification: The extracted "hypotheses" (product-attribute-value relations) are then linked to a structured product catalog. The system checks for exact matches or uses GloVe embeddings for fuzzy matching of attribute names. If all hypotheses can be verified against the catalog, the statement is deemed factual.
A key innovation is the development of a set of concise, domain-specific rules derived from dependency trees. The authors also explore DepCheckerGA, a genetic optimization approach to automatically learn these rules, aiming to reduce manual effort.
Datasets
The authors developed a BART-based NLG model to generate conversational statements for the e-commerce domain. This model was trained on human-written descriptions of product attribute values from a structured product catalog. The generated sentences from this NLG model served as the test set for evaluating DepChecker's factuality assessment capabilities. The paper also references the TabFact dataset as a benchmark for general-purpose table-based fact verification, noting that their specialized approach is needed because general models don't perform well on their domain-specific data.
Key Insights and Ideas
- Domain Specificity Matters: General-purpose fact-checking models, even when fine-tuned, fail to perform adequately on the specific characteristics of NLG-generated text in conversational e-commerce. This highlights the need for tailored solutions.
- Dependency Trees as a Powerful Feature: Dependency parse structures provide a robust way to extract claims and relationships from text, especially when the language style is relatively consistent.
- Speed and Explainability are Crucial: For real-time conversational systems, speed is paramount. DepChecker's rule-based approach offers significant speed advantages over large neural models and provides explainability, allowing for easier debugging and understanding of errors.
- Unsupervised Approach Advantage: The unsupervised nature of DepChecker (once rules are defined) reduces the need for large amounts of labeled data, which can be costly and time-consuming to acquire for specific domains.
Key Results
- Accuracy: DepChecker achieved an accuracy of 65% and a false recall of 94%, performing comparably to the neural baseline RoBERTaChecker (67% accuracy, 94% false recall) but with significant speed benefits.
- Speed: DepChecker was 3-10 times faster than other baseline models and 9.4 times faster than RoBERTaChecker. The average inference time, including dependency parsing, was around 72 milliseconds per sentence.
- DepCheckerGA: The automatically learned rules via genetic optimization achieved statistically indistinguishable performance from manually created rules but required almost twice as many rules, leading to a higher average inference time (around 250ms).
- Error Analysis: DepChecker primarily makes errors when attribute mentions are slightly altered (e.g., "soft material" for "soft") or when it misinterprets the head of a phrase. Neural models like RoBERTaChecker made different types of errors, such as misclassifying negated attributes or associating values with the wrong product.
Limitations and Future Work
- Manual Rule Creation Effort: The primary limitation acknowledged is the manual effort required to construct the initial set of rules for DepChecker. While DepCheckerGA automates this, it leads to less compact and efficient rules.
- Generalizability to High Textual Variation: The current system is optimized for the relatively consistent linguistic style of conversational shopping. Extending it to domains with higher textual variation is a direction for future work.
- Handling Aggregations: DepChecker, in its basic form, struggles with complex aggregations in structured data (like those found in the TabFact dataset).
- Future Work: The authors suggest further exploring DepCheckerGA's rule combining strategies, investigating multiple sequence alignment techniques for better rule generalization, and extending the system to handle more complex fact-checking scenarios and domains with greater linguistic diversity. They also propose that the speed and explainability of rule-based systems like DepChecker make them valuable for production use cases, especially as larger, slower models become more prevalent.
Summary updated