Textual Paraphrase Dataset for Deep Language Modelling

Jenna Kanerva, Filip Ginter, Li-Hsin Chang, Valtteri Skantsi, Jemina Kilpeläinen, Hanna-Mari Kupari, Aurora Piirto, Jenna Saarni, Maija Sevón, Otto TarkkaCognitive 2022 · DOI · OpenAlex

Abstract

The Turku Paraphrase Corpus is a dataset of over 100,000 Finnish paraphrase pairs. During the corpus creation, we strived to gather challenging paraphrase pairs, more suitable to test the capabilities of natural language understanding models. The paraphrases are both selected and classified manually, so as to minimise lexical overlap, and provide examples that are structurally and lexically different to the maximum extent. An important distinguishing feature of the corpus is that most of the paraphrase pairs are extracted and distributed in their native document context, rather than in isolation. The primary application for the dataset is the development and evaluation of deep language models, and representation learning in general.

Introduction

Deep learning models have achieved remarkable success in supervised Natural Language Processing (NLP) tasks. However, their ability to perform true "representation learning"—understanding semantic equivalence regardless of surface-level wording—remains a challenge. Many existing paraphrase datasets are generated automatically through methods like back-translation, which often results in high lexical overlap. This "lexical bias" allows models to rely on word-level shortcuts rather than deep semantic understanding. Furthermore, the common practice of presenting paraphrase pairs in isolation strips away the document context that often defines semantic equivalence.

The Turku Paraphrase Corpus addresses these issues for Finnish, an under-resourced language. By employing a manual extraction and classification process, the corpus provides a high-quality, context-aware resource designed to push the boundaries of Natural Language Understanding (NLU).

Methodology

The corpus was developed using a human-in-the-loop approach involving six native Finnish annotators. The process was divided into two primary stages: manual extraction and classification.

Manual Extraction and Classification

Annotators compared side-by-side documents (e.g., different translations of the same film) to identify paraphrase candidates. Unlike automated systems, humans can identify "hard" paraphrases with minimal word overlap. Each pair was then classified on a 4-point scale:

  • Label 4: Universal paraphrases (equivalent in all contexts).
  • Label 3: Context-dependent paraphrases.
  • Label 2: Related but not paraphrases.
  • Label 1: Unrelated.

Annotators also used optional flags for Label 4 pairs to indicate directional subsumption, style differences, or minor grammatical variations. If a pair was nearly a perfect paraphrase, annotators were encouraged to rewrite them into "Label 4" versions (denoted as Label 4</> in the data).

Distribution of paraphrase labels across the six annotators, showing the prevalence of high-quality Label 3 and Label 4 pairs.

Dataset Characteristics and Results

The resulting Turku Paraphrase Corpus contains 104,645 manually classified pairs, making it one of the largest manually annotated paraphrase resources globally. Of these, 91,604 are naturally occurring pairs, while 13,041 were produced through manual rewriting. The primary data sources include movie subtitles (~80%), news headings, and discussion forums.

Annotator Performance and Diversity

The project achieved an inter-annotator agreement of approximately 70%, which is competitive with major English benchmarks. Interestingly, extraction rates varied significantly between annotators, ranging from 13 to 50 pairs per document. This variation is viewed as a strength, as it introduces a broader diversity of linguistic examples into the dataset.

Detailed extraction statistics per annotator, highlighting the scale of the manual effort and the variation in extraction density.

Key Insights

  • Contextual Importance: By distributing pairs within their native document context, the corpus enables research into how surrounding text influences meaning—a first for paraphrase datasets.
  • Efficiency: The manual extraction method proved highly efficient; 98% of the final dataset consists of positive examples (Labels 3 and 4), avoiding the noise typical of random sampling.
  • Directionality: The inclusion of subsumption flags acknowledges that semantic equivalence is not always symmetric, providing a more nuanced view of linguistic relationships.

Conclusion and Future Work

While the dataset is currently concentrated in the domain of movie subtitles, it provides a robust foundation for Finnish NLU. Future work involves leveraging the document context for tasks such as "paraphrase span detection," where models must locate a paraphrase within a larger text body. The Turku Paraphrase Corpus is available under a CC-BY-SA license, supporting open research in deep language modelling.

Summary updated

Abstract

Abstract The Turku Paraphrase Corpus is a dataset of over 100,000 Finnish paraphrase pairs. During the corpus creation, we strived to gather challenging paraphrase pairs, more suitable to test the capabilities of natural language understanding models. The paraphrases are both selected and classified manually, so as to minimise lexical overlap, and provide examples that are structurally and lexically different to the maximum extent. An important distinguishing feature of the corpus is that most of the paraphrase pairs are extracted and distributed in their native document context, rather than in isolation. The primary application for the dataset is the development and evaluation of deep language models, and representation learning in general.
BibTeX
@inproceedings{kanerva2022textual,
  author = {Jenna Kanerva and Filip Ginter and Li-Hsin Chang and Valtteri Skantsi and Jemina Kilpeläinen and Hanna-Mari Kupari and Aurora Piirto and Jenna Saarni and Maija Sevón and Otto Tarkka},
  title = {{Textual Paraphrase Dataset for Deep Language Modelling}},
  booktitle = {Cognitive},
  year = {2022},
  doi = {10.1007/978-3-031-17258-8\_27},
}
Figure