Abstract
Conversational machine reading systems help users answer high-level questions (e.g. determine if they qualify for particular government benefits) when they do not know the exact rules by which the determination is made (e.g. whether they need certain income levels or veteran status). The key challenge is that these rules are only provided in the form of a procedural text (e.g. guidelines from government website) which the system must read to figure out what to ask the user. We present a new conversational machine reading model that jointly extracts a set of decision rules from the procedural text while reasoning about which are entailed by the conversational history and which still need to be edited to create questions for the user. On the recently introduced ShARC conversational machine reading dataset, our Entailment-driven Extract and Edit network (E3) achieves a new state-of-the-art, outperforming existing systems as well as a new BERT-based baseline. In addition, by explicitly highlighting which information still needs to be gathered, E3 provides a more explainable alternative to prior work.
Introduction
In Conversational Machine Reading (CMR), a system acts as an intermediary between a user and a complex document of regulations or guidelines. Unlike standard span-based Question Answering, CMR requires a multi-turn dialogue. The system must identify which rules apply to the user's specific situation, determine which criteria the user has already satisfied (based on a provided "scenario" or previous answers), and formulate natural follow-up questions to clarify missing information.
The core difficulty lies in the "gap" between raw regulatory text and a helpful conversation. Previous systems often relied on hand-engineered pipelines or black-box sequence-to-sequence models that struggled to reason explicitly about which rules were still "active" or required further inquiry.
The E3 Architecture
The authors propose the $E^3$ network (Entailment-driven Extracting and Editing), which integrates extraction, reasoning, and generation into a unified framework. The model is composed of three primary modules that work in tandem to process the input $x$, which includes the question, rule text, scenario, and dialogue history.
1. Extraction Module
Using a BERT Transformer encoder, the model identifies spans in the rule text that represent "latent rules" or decision criteria (e.g., "UK resident"). It treats this as a span-prediction task, identifying start and end points for potential rules $R_1 ... R_{n_R}$. This ensures the model's reasoning is grounded directly in the source document.
2. Entailment Module
This module serves as the system's reasoning engine. For every extracted rule, it calculates an entailment score ($g_i, h_i$) based on the user's initial scenario and the history of previous interactions. By explicitly scoring each rule's fulfillment, the model tracks which parts of the policy have been satisfied and which remain unknown, preventing redundant or irrelevant questions.
3. Decision & Editor Modules
The Decision module aggregates the entailment scores to determine the next action: providing a final answer (Yes, No, or Irrelevant) or deciding to Inquire further. If an inquiry is necessary, the Editor module (an attentive decoder) takes the raw text span of an un-entailed rule and "edits" it into a fluent, natural language question. For example, it might transform the rule span "UK resident" into the question "Are you a UK resident?"
Experimental Results
The model was evaluated on the ShARC (Shaping Answers with Rules through Conversation) benchmark, which contains 32k utterances based on 630 real-world procedural documents. $E^3$ achieved a new state-of-the-art performance across several metrics:
- Decision Accuracy: The model achieved 67.6% micro-accuracy, significantly outperforming the previous best "Pipeline" model (61.9%) and a strong BERT-based QA baseline (63.6%).
- Question Quality: Follow-up questions generated by the Editor module achieved a 38.7 BLEU4 score, demonstrating superior fluency and relevance compared to earlier methods.
- Ablation Insights: Removing either the Editor or the Entailment module led to significant performance drops, confirming that both explicit tracking of satisfied rules and fluent rephrasing are vital for effective CMR.
Explainability and Conclusion
A key advantage of $E^3$ is its explainability by design. Because the model explicitly extracts rules and scores their entailment, developers and users can visualize exactly which part of a document the model is focused on and why it chose a specific follow-up question. This transparency is a significant improvement over previous black-box approaches.
While $E^3$ represents a major advancement, the authors note that challenges remain in handling complex numerical reasoning (e.g., income thresholds) and resolving ambiguities in hierarchical document structures. Future work will focus on more sophisticated entailment models and improving the robustness of the span extraction process.
Summary updated