Introduction
As deep learning models are increasingly integrated into critical decision-making processes, the demand for Explainable AI (XAI) has intensified. While traditional methods like saliency maps or attention weights offer some transparency, they often lack the interpretability required by end-users. Natural language explanations provide a more intuitive bridge between model logic and human understanding. However, two primary obstacles prevent the widespread adoption of sentence-level explanations: high computational latency and the "data hunger" of supervised models.
Standard explanation models are typically autoregressive, generating text one token at a time. This sequential process introduces significant delays, often making the explanation arrive much later than the prediction itself. Furthermore, training these models usually requires vast amounts of human-annotated rationales, which are costly and difficult to obtain in niche or emerging domains. To address these challenges, this paper introduces the Classification Non-Autoregressive Transformer (C-NAT), a framework designed for high-speed, low-resource explanation generation.
The C-NAT Framework
The C-NAT architecture rethinks the explanation task by moving away from sequential generation in favor of a parallel approach. The framework consists of several key components:
Parallel Generation via Fertility Prediction
At the heart of C-NAT is a Fertility Predictor. Instead of waiting for the previous word to determine the next, the model predicts a "fertility" score for each input token, which dictates how many times that token should be copied into the decoder's initial state. This creates a skeleton for the explanation, allowing the decoder to generate all words simultaneously. This shift from $O(N)$ to $O(1)$ time complexity for the length of the explanation is the primary driver of the model's efficiency.
Simultaneous Classification and Explanation
C-NAT treats classification and explanation as a joint task. A dedicated Label Predictor utilizes the hidden states from the decoder to output the final classification label at the same time the explanation is being formed. This multi-task learning approach ensures that the generated explanation is grounded in the features used for the actual prediction.
Ensuring Fluency with LM Discrimination
Non-autoregressive models often suffer from "word salad" or repetitive outputs because they lack the conditional dependency between adjacent tokens. To mitigate this, the authors incorporate a pre-trained Language Model (LM) Discriminator. During training, this discriminator evaluates the fluency of the generated sequences, forcing the model to produce grammatically coherent and human-readable explanations without sacrificing inference speed.
Addressing Data Scarcity
A significant contribution of this work is its ability to function in "low-resource" environments where human explanations are scarce or non-existent.
- Weakly-Supervised Learning: Using a technique called "data programming," the authors start with a tiny seed of just 30 human-annotated explanations. These are used to create labeling functions and templates that automatically generate a large "pseudo-labeled" dataset, providing enough signal to train the C-NAT model effectively.
- Unsupervised Learning: In scenarios with zero human rationales, the framework employs a back-translation mechanism. By paraphrasing input sentences into pseudo-explanations, the model learns to extract and rephrase relevant information into an explanatory format without ever seeing a human-written example.
Experimental Evaluation
The researchers evaluated C-NAT across several benchmarks:
- e-SNLI: A large-scale Natural Language Inference dataset containing human-written explanations for entailment, contradiction, and neutral labels.
- Spouse Prediction (SP): A relation extraction task used to test the weakly-supervised approach, starting with only 30 human samples.
- SNLI-Pseudo: A variant of the SNLI dataset used to validate the unsupervised back-translation strategy.
Key Results
The experimental results highlight the efficiency and effectiveness of the C-NAT framework:
- Inference Speedup: C-NAT achieved a staggering 21.40x speedup over autoregressive baselines. On the NLI task, latency was reduced from approximately 1000ms to just 47ms, making real-time explanation feasible.
- Improved Accuracy: Surprisingly, C-NAT often outperformed models that did not generate explanations. In the supervised NLI task, C-NAT reached 85.23% accuracy, compared to 79.46% for a standard Transformer baseline. This suggests that the multi-task learning objective helps the model learn better internal representations.
- Robustness in Low-Resource Settings: Even in the unsupervised setting, the model achieved a classification accuracy of 83.12%. In the weakly-supervised Spouse Prediction task, it maintained high rationality scores while operating significantly faster than traditional models.
Conclusion and Future Work
The C-NAT framework demonstrates that the trade-off between model interpretability and performance is not inevitable. By leveraging non-autoregressive generation and innovative low-resource training strategies, the authors have created a system that is both fast enough for production environments and flexible enough for domains where data is scarce.
While the LM discriminator significantly improves fluency, future work may focus on further closing the gap between non-autoregressive and autoregressive models regarding complex syntactic logic. Nonetheless, C-NAT represents a major step toward making "Explainable AI" a practical reality for real-world applications.
Summary updated