Parallel Sentence-Level Explanation Generation for Real-World Low-Resource Scenarios

Yan Liu, Xiaokang Chen, Qi Dai

Introduction

As deep learning models are increasingly integrated into critical decision-making processes, the demand for Explainable AI (XAI) has intensified. While traditional methods like saliency maps or attention weights offer some transparency, they often lack the interpretability required by end-users. Natural language explanations provide a more intuitive bridge between model logic and human understanding. However, two primary obstacles prevent the widespread adoption of sentence-level explanations: high computational latency and the "data hunger" of supervised models.

Standard explanation models are typically autoregressive, generating text one token at a time. This sequential process introduces significant delays, often making the explanation arrive much later than the prediction itself. Furthermore, training these models usually requires vast amounts of human-annotated rationales, which are costly and difficult to obtain in niche or emerging domains. To address these challenges, this paper introduces the Classification Non-Autoregressive Transformer (C-NAT), a framework designed for high-speed, low-resource explanation generation.

The C-NAT Framework

The C-NAT architecture rethinks the explanation task by moving away from sequential generation in favor of a parallel approach. The framework consists of several key components:

Parallel Generation via Fertility Prediction

At the heart of C-NAT is a Fertility Predictor. Instead of waiting for the previous word to determine the next, the model predicts a "fertility" score for each input token, which dictates how many times that token should be copied into the decoder's initial state. This creates a skeleton for the explanation, allowing the decoder to generate all words simultaneously. This shift from $O(N)$ to $O(1)$ time complexity for the length of the explanation is the primary driver of the model's efficiency.

Simultaneous Classification and Explanation

C-NAT treats classification and explanation as a joint task. A dedicated Label Predictor utilizes the hidden states from the decoder to output the final classification label at the same time the explanation is being formed. This multi-task learning approach ensures that the generated explanation is grounded in the features used for the actual prediction.

Ensuring Fluency with LM Discrimination

Non-autoregressive models often suffer from "word salad" or repetitive outputs because they lack the conditional dependency between adjacent tokens. To mitigate this, the authors incorporate a pre-trained Language Model (LM) Discriminator. During training, this discriminator evaluates the fluency of the generated sequences, forcing the model to produce grammatically coherent and human-readable explanations without sacrificing inference speed.

Addressing Data Scarcity

A significant contribution of this work is its ability to function in "low-resource" environments where human explanations are scarce or non-existent.

  1. Weakly-Supervised Learning: Using a technique called "data programming," the authors start with a tiny seed of just 30 human-annotated explanations. These are used to create labeling functions and templates that automatically generate a large "pseudo-labeled" dataset, providing enough signal to train the C-NAT model effectively.
  2. Unsupervised Learning: In scenarios with zero human rationales, the framework employs a back-translation mechanism. By paraphrasing input sentences into pseudo-explanations, the model learns to extract and rephrase relevant information into an explanatory format without ever seeing a human-written example.

Experimental Evaluation

The researchers evaluated C-NAT across several benchmarks:

  • e-SNLI: A large-scale Natural Language Inference dataset containing human-written explanations for entailment, contradiction, and neutral labels.
  • Spouse Prediction (SP): A relation extraction task used to test the weakly-supervised approach, starting with only 30 human samples.
  • SNLI-Pseudo: A variant of the SNLI dataset used to validate the unsupervised back-translation strategy.

Key Results

The experimental results highlight the efficiency and effectiveness of the C-NAT framework:

  • Inference Speedup: C-NAT achieved a staggering 21.40x speedup over autoregressive baselines. On the NLI task, latency was reduced from approximately 1000ms to just 47ms, making real-time explanation feasible.
  • Improved Accuracy: Surprisingly, C-NAT often outperformed models that did not generate explanations. In the supervised NLI task, C-NAT reached 85.23% accuracy, compared to 79.46% for a standard Transformer baseline. This suggests that the multi-task learning objective helps the model learn better internal representations.
  • Robustness in Low-Resource Settings: Even in the unsupervised setting, the model achieved a classification accuracy of 83.12%. In the weakly-supervised Spouse Prediction task, it maintained high rationality scores while operating significantly faster than traditional models.

Conclusion and Future Work

The C-NAT framework demonstrates that the trade-off between model interpretability and performance is not inevitable. By leveraging non-autoregressive generation and innovative low-resource training strategies, the authors have created a system that is both fast enough for production environments and flexible enough for domains where data is scarce.

While the LM discriminator significantly improves fluency, future work may focus on further closing the gap between non-autoregressive and autoregressive models regarding complex syntactic logic. Nonetheless, C-NAT represents a major step toward making "Explainable AI" a practical reality for real-world applications.

Summary updated

Abstract

In order to reveal the rationale behind model predictions, many works have exploited providing explanations in various forms. Recently, to further guarantee readability, more and more works turn to generate sentence-level human language explanations. However, current works pursuing sentence-level explanations rely heavily on annotated training data, which limits the development of interpretability to only a few tasks. As far as we know, this paper is the first to explore this problem smoothly from weak-supervised learning to unsupervised learning. Besides, we also notice the high latency of autoregressive sentence-level explanation generation, which leads to asynchronous interpretability after prediction. Therefore, we propose a non-autoregressive interpretable model to facilitate parallel explanation generation and simultaneous prediction. Through extensive experiments on Natural Language Inference task and Spouse Prediction task, we find that users are able to train classifiers with comparable performance $10-15\times$ faster with parallel explanation generation using only a few or no annotated training data.
BibTeX
@inproceedings{liu2023parallel,
  author = {Yan Liu and Xiaokang Chen and Qi Dai},
  title = {{Parallel Sentence-Level Explanation Generation for Real-World Low-Resource Scenarios}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
}
Figure