Modelling Sentence Pairs via Reinforcement Learning: An Actor-Critic Approach to Learn the Irrelevant Words

Mahtab Ahmed, Robert E. MercerAAAI 2020 · DOI

Introduction

In Natural Language Processing (NLP), comparing sentence pairs—such as identifying paraphrases or determining logical entailment—typically requires encoding every word into high-dimensional vectors. However, human intuition suggests that not all words are equally important. Often, filler phrases and stop words introduce noise that can obscure the core semantic relationship between two sentences. This paper introduces a reinforcement learning (RL) framework designed to automatically identify and remove task-specific "irrelevant" words, creating condensed and more effective sentence representations.

Methodology: The Actor-Critic Framework

The authors propose an Actor-Critic architecture to mimic the human process of "trimming the fat" from text.

  • The Actor (The Pruner): For every word in a sentence, the actor makes a binary decision: Keep or Delete. Crucially, this decision is context-dependent. The actor considers the current word, the previous hidden state of an LSTM, and a summary of the entire counterpart sentence. This allows the model to recognize that a word's relevance is often determined by what is present in the other sentence.
  • The Critic (The Evaluator): Based on the InferSent architecture (a standard bi-LSTM), the critic evaluates how well the pruned sentence pair performs on the specific task (e.g., classification accuracy).
  • Reward and Stability: The actor receives rewards based on the critic's performance. To prevent degenerate solutions (like deleting everything), a unimodal reward function penalizes the model if the deletion rate falls outside a target range. Additionally, the authors implement a delayed update mechanism to stabilize training, preventing the model from oscillating wildly during the RL process.

Experimental Results

The model was evaluated on three major benchmarks: the Microsoft Research Paraphrase Corpus (MSRP), the Sentences Involving Compositional Knowledge (SICK) dataset, and the AI2-8grade science question dataset.

As shown in the results table below, the "InferSent + RL" ensemble achieved state-of-the-art performance, outperforming several complex tree-based and attention-based architectures. Notably, it reached 76.12% accuracy on MSRP and 86.12% on SICK.

Table 1: Performance comparison of the proposed RL model against various baseline and state-of-the-art models across three datasets.

Qualitative Analysis: Explainability through Deletion

A key advantage of this approach is its inherent explainability. By observing which words the actor chooses to delete, researchers can see exactly what the model considers "noise."

Table 2: Examples of sentence pairs from different datasets with irrelevant words crossed out by the actor.

In the examples above, the model successfully identifies and removes filler phrases like "a piece of" or "to the section that" without losing the core meaning. On the AI2-8grade dataset, the model was able to remove roughly 90% of the words while maintaining competitive performance, demonstrating that a vast majority of text in certain tasks is redundant. Furthermore, the model learns context-specific irrelevance; for instance, it may discard color adjectives in the SICK dataset if they do not impact the logical entailment between the two sentences.

Conclusion and Future Work

By framing sentence pruning as a reinforcement learning problem, this research demonstrates that "less is more" in sentence pair modeling. The actor-critic approach effectively filters noise, leading to improved accuracy and more interpretable models. While the current implementation uses LSTMs, future work could involve integrating this deletion policy into Transformer-based architectures like BERT or RoBERTa to improve the efficiency of self-attention mechanisms by pruning irrelevant tokens early in the pipeline.

Summary updated

BibTeX
@inproceedings{ahmed2020modelling,
  author = {Mahtab Ahmed and Robert E. Mercer},
  title = {{Modelling Sentence Pairs via Reinforcement Learning: An Actor-Critic Approach to Learn the Irrelevant Words}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i05.6230},
}
Figure