Introduction
In Natural Language Processing (NLP), comparing sentence pairs—such as identifying paraphrases or determining logical entailment—typically requires encoding every word into high-dimensional vectors. However, human intuition suggests that not all words are equally important. Often, filler phrases and stop words introduce noise that can obscure the core semantic relationship between two sentences. This paper introduces a reinforcement learning (RL) framework designed to automatically identify and remove task-specific "irrelevant" words, creating condensed and more effective sentence representations.
Methodology: The Actor-Critic Framework
The authors propose an Actor-Critic architecture to mimic the human process of "trimming the fat" from text.
- The Actor (The Pruner): For every word in a sentence, the actor makes a binary decision: Keep or Delete. Crucially, this decision is context-dependent. The actor considers the current word, the previous hidden state of an LSTM, and a summary of the entire counterpart sentence. This allows the model to recognize that a word's relevance is often determined by what is present in the other sentence.
- The Critic (The Evaluator): Based on the InferSent architecture (a standard bi-LSTM), the critic evaluates how well the pruned sentence pair performs on the specific task (e.g., classification accuracy).
- Reward and Stability: The actor receives rewards based on the critic's performance. To prevent degenerate solutions (like deleting everything), a unimodal reward function penalizes the model if the deletion rate falls outside a target range. Additionally, the authors implement a delayed update mechanism to stabilize training, preventing the model from oscillating wildly during the RL process.
Experimental Results
The model was evaluated on three major benchmarks: the Microsoft Research Paraphrase Corpus (MSRP), the Sentences Involving Compositional Knowledge (SICK) dataset, and the AI2-8grade science question dataset.
As shown in the results table below, the "InferSent + RL" ensemble achieved state-of-the-art performance, outperforming several complex tree-based and attention-based architectures. Notably, it reached 76.12% accuracy on MSRP and 86.12% on SICK.
Qualitative Analysis: Explainability through Deletion
A key advantage of this approach is its inherent explainability. By observing which words the actor chooses to delete, researchers can see exactly what the model considers "noise."
In the examples above, the model successfully identifies and removes filler phrases like "a piece of" or "to the section that" without losing the core meaning. On the AI2-8grade dataset, the model was able to remove roughly 90% of the words while maintaining competitive performance, demonstrating that a vast majority of text in certain tasks is redundant. Furthermore, the model learns context-specific irrelevance; for instance, it may discard color adjectives in the SICK dataset if they do not impact the logical entailment between the two sentences.
Conclusion and Future Work
By framing sentence pruning as a reinforcement learning problem, this research demonstrates that "less is more" in sentence pair modeling. The actor-critic approach effectively filters noise, leading to improved accuracy and more interpretable models. While the current implementation uses LSTMs, future work could involve integrating this deletion policy into Transformer-based architectures like BERT or RoBERTa to improve the efficiency of self-attention mechanisms by pruning irrelevant tokens early in the pipeline.
Summary updated