Abstract
Conversational machine reading systems help users answer high-level questions when they do not know the exact rules by which the determination is made. The key challenge is that these rules are only provided in the form of a procedural text which the system must read to figure out what to ask the user. We present a new conversational machine reading model, the Entailment-driven Extract and Edit network (E3), that jointly extracts a set of decision rules from the procedural text while reasoning about which are entailed by the conversational history and which still need to be edited to create questions for the user. On the ShARC dataset, E3 achieves a new state-of-the-art, outperforming existing systems and providing a more explainable alternative to prior work.
Introduction
Conversational Machine Reading (CMR) involves systems that can navigate complex regulatory or procedural texts to help users reach a decision. For instance, a user might ask, "Do I need to pay tax on my pension?" based on a government guideline. To provide an answer, the system must identify relevant rules within the text, determine which rules the user has already satisfied through their initial scenario or previous dialogue, and formulate follow-up questions for any remaining unknowns.
Previous approaches often treated CMR as a "black-box" sequence-to-sequence task, which can lead to redundant questions or "hallucinations" not grounded in the source text. We propose the E3 network to address these issues through explicit rule extraction and entailment reasoning.
The E3 Architecture
The E3 network decomposes the CMR task into three interpretable modules: Extraction, Entailment, and Decision.
Extraction Module
Using a BERT-based transformer encoder, the model identifies potential "latent rules" within the document. These are spans of text that represent conditions for a decision (e.g., "UK resident").
Entailment Module
This module acts as the reasoning engine. It calculates entailment scores between the extracted rules and the user's provided scenario and history. By determining which rules are already satisfied, the model avoids redundant questioning.
Decision and Editor Module
The system decides if it can provide a final answer ("Yes," "No," or "Irrelevant"). If more information is required ("Inquire"), it selects the most relevant un-entailed rule. An LSTM-based Editor then transforms the raw text span into a natural, fluent question for the user.
Experimental Results
We evaluated E3 on the ShARC (Shaping Answers for Real-world Conversations) dataset, which contains 32k utterances based on 630 regulatory snippets. E3 achieved state-of-the-art performance:
- Decision Accuracy: 67.6% micro-accuracy, significantly higher than the previous best pipeline (61.9%).
- Question Quality: A BLEU4 score of 38.7 for follow-up questions, improving upon the previous benchmark of 34.4.
Ablation studies confirmed that both the Editor and Entailment modules are critical for performance, ensuring the model remains grounded in the text while reasoning effectively over the conversation.
Discussion and Limitations
E3 provides a more explainable framework by pointing to specific text spans used for decision-making. However, challenges remain. The model struggles with numerical logic (e.g., income thresholds) and lacks broad world knowledge (e.g., knowing that living in Paris implies not living in the UK). Future work will focus on integrating more robust logical reasoning and external knowledge into the architecture.
Source code available at: https://github.com/vzhong/e3
Summary updated