Convolutional Neural Network for Paraphrase Identification

Wenpeng Yin, Hinrich Schütze

Abstract

Paraphrase Identification (PI) is a fundamental natural language processing task that involves determining whether two sentences convey the same meaning. Traditional approaches often struggle with the varying levels of linguistic granularity at which paraphrastic relationships occur. This paper presents the Bi-CNN-MI architecture, which captures interactions between sentences at four distinct levels: unigrams, short n-grams, long n-grams, and full sentences. By combining a Siamese convolutional neural network with a novel unsupervised pretraining strategy, the proposed model achieves state-of-the-art results on the Microsoft Research Paraphrase (MSRP) corpus.

Introduction

The core challenge in Paraphrase Identification lies in the diversity of semantic equivalence. In some cases, a single word difference (e.g., "ten percent" vs. "five percent") can negate a paraphrase. In others, two sentences may share almost no lexical overlap yet remain semantically identical (e.g., "Mary gave birth to a son in 2000" vs. "He is 14 years old and his mother is Mary").

Existing models typically focus on either fine-grained word overlaps or coarse-grained sentence-level embeddings. However, a robust system must analyze interactions across multiple levels of granularity simultaneously. This paper addresses this need by introducing a multigranular interaction model.

Methodology: The Bi-CNN-MI Architecture

The authors propose Bi-CNN-MI (Bi-Convolutional Neural Network with Multigranular Interaction), a framework designed to "digest" sentences and compare them at multiple stages of abstraction.

The Bi-CNN-MI architecture showing multigranular feature extraction and interaction.

The architecture consists of three primary components:

1. Sentence Analysis (CNN-SM)

A Siamese architecture employs two identical CNNs to process each sentence. These networks utilize "wide convolution" and "k-max pooling" to extract features. Unlike standard CNNs that output a single vector, CNN-SM preserves intermediate representations at four levels: unigrams (words), short n-grams, long n-grams, and the full sentence representation.

2. Interaction Model (CNN-IM)

Rather than comparing only the final sentence vectors, the Interaction Model constructs "interaction matrices" for every level of granularity. It calculates the similarity between every word or phrase in Sentence A and every corresponding unit in Sentence B. This allows the model to capture subtle overlaps that might be lost in a global sentence embedding.

3. Dynamic Pooling and Classification

Because sentences vary in length, the resulting interaction matrices are of different sizes. A dynamic pooling strategy is employed to map these matrices into fixed-size feature vectors. These vectors are then concatenated and fed into a logistic regression classifier to produce the final "Yes/No" paraphrase prediction.

Unsupervised Pretraining (CNN-LM)

Deep learning models for PI often suffer from data scarcity, as benchmark datasets like MSRP are relatively small. To mitigate this, the authors developed an unsupervised pretraining task using Noise Contrastive Estimation (NCE).

The unsupervised pretraining scheme using Noise Contrastive Estimation (NCE).

The CNN is trained to predict a target word based on the representation of its surrounding context. Using 100,000 unlabeled sentences from the English Gigaword corpus, the network learns the underlying structure of language before being fine-tuned on the paraphrase task. This "warm start" is critical for initializing the CNN filters effectively and preventing overfitting on the small MSRP dataset.

Experimental Results

The Bi-CNN-MI model was evaluated on the MSRP corpus, the standard benchmark for paraphrase identification. It outperformed previous neural network competitors, including Recursive Autoencoders (RAE):

  • Accuracy: 78.1% (compared to 76.7% for RAE).
  • F1 Score: 84.4 (compared to 83.6 for RAE).

Ablation studies revealed that n-gram levels (short and long phrases) were the most informative features. Interestingly, unigram features alone contributed the least, suggesting that paraphrases rely more on semantic concepts than exact word matches. Furthermore, the unsupervised pretraining proved essential; without it, accuracy dropped significantly to 72.5%.

Conclusion

The Bi-CNN-MI architecture demonstrates that multigranular interaction is a powerful approach for capturing semantic equivalence. By comparing sentences at various levels of abstraction and leveraging unsupervised pretraining, the model achieves superior performance without relying on formal grammar parsers. This makes it particularly robust for processing noisy text. Future work may extend this multigranular approach to other text-matching domains, such as Question Answering and Information Retrieval.

Summary updated

Abstract

We present a new deep learning architecture Bi-CNN-MI for paraphrase identification (PI). Based on the insight that PI requires comparing two sentences on multiple levels of granularity, we learn multigranular sentence representations using convolutional neural network (CNN) and model interaction features at each level. These features are then the input to a logistic classifier for PI. All parameters of the model (for embeddings, convolution and classification) are directly optimized for PI. To address the lack of training data, we pretrain the network in a novel way using a language modeling task. Results on the MSRP corpus surpass that of previous NN competitors.
BibTeX
@inproceedings{yin2015convolutional,
  author = {Wenpeng Yin and Hinrich Schütze},
  title = {{Convolutional Neural Network for Paraphrase Identification}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
  doi = {10.3115/v1/n15-1091},
}
Figure