Abstract
Taxonomies serve as critical structural frameworks for various Natural Language Processing (NLP) tasks, yet they often suffer from low recall due to the limited linguistic variety of manually curated terms. This paper investigates the task of taxonomy augmentation through automatic paraphrase generation. We compare a traditional statistical machine translation (SMT) system, Moses, against a neural sequence-to-sequence (seq2seq) model. Our findings reveal that while neural models exhibit greater lexical diversity, the statistical approach is significantly more reliable for domain-specific augmentation, successfully expanding a risk-domain taxonomy's coverage by 22%.
Introduction
Building a comprehensive taxonomy is a "Goldilocks" problem: small taxonomies lack the coverage necessary for high-recall tasks, while manual expansion is prohibitively labor-intensive. Even automatically mined taxonomies often fail to capture the myriad ways humans express the same concept. For instance, a system might recognize "trafficked people" but fail to identify "trafficking in persons" unless both are explicitly mapped.
This study addresses taxonomy augmentation by framing paraphrase generation as a translation task—essentially translating English into alternative English. The goal is to take an existing list of terms and generate valid synonyms or paraphrases that can be used to improve document classification and search retrieval.
Methodology
The researchers evaluated two distinct architectural approaches to paraphrase generation:
- Moses (Statistical Machine Translation): A phrase-based SMT system utilizing a tri-gram language model and a translation table to map phrases based on statistical likelihood.
- Seq2seq (Neural Machine Translation): A deep learning architecture consisting of a bi-directional LSTM encoder and an LSTM decoder with an attention mechanism, representing the state-of-the-art in neural translation at the time of the study.
To prevent the models from simply memorizing test data, a graph-based splitting method was employed. Phrases were treated as nodes and paraphrase pairs as edges; entire connected components were then assigned exclusively to either the training or test set.
Datasets
Models were trained on a filtered subset of PPDB 2.0 (The Paraphrase Database), specifically the "large pack" filtered for equivalent pairs, resulting in ~757,000 training points. Evaluation was conducted on a Risk Taxonomy containing 2,824 terms and a News Archive of 14 million articles to measure real-world coverage improvements.
Results and Analysis
Quantitative Performance
In a surprising departure from recent trends in neural NLP, the statistical Moses model significantly outperformed the neural seq2seq model. Moses achieved a BLEU score of 0.4098, compared to 0.3156 for seq2seq. Human evaluation of the top-generated paraphrases confirmed this gap in precision.
As shown in the table above, Moses produced 1,337 valid new terms, whereas the seq2seq model produced only 419. Furthermore, seq2seq generated a staggering 2,042 invalid paraphrases, suggesting that while the neural model is more "creative," it lacks the constraint necessary for high-precision taxonomy work.
Qualitative Comparison
The models' differing behaviors are best illustrated through direct examples. While Moses tends to produce conservative, high-fidelity synonyms, seq2seq often generates related but semantically distinct concepts that are unsuitable for a taxonomy.
For example, for the term "unexpected entry of competitors," Moses suggested "unpredicted entrance of competitors," which preserves the original meaning. In contrast, seq2seq suggested "accident," which is too broad and loses the specific context of market competition.
Discussion and Conclusion
The primary hurdle for both models remains domain specificity. Because the models were trained on general-purpose data (PPDB), they occasionally struggle with the nuances of specialized fields. For instance, replacing "concentrations" with "levels" is valid in a chemical context but incorrect when referring to "sector concentrations" in finance.
Despite these challenges, the Moses-augmented taxonomy increased coverage of the news corpus by 22%, compared to a mere 7.6% boost from seq2seq. This study highlights that for specialized, high-precision tasks like taxonomy augmentation, traditional statistical methods may still offer superior performance over more complex neural architectures, particularly when domain-specific training data is scarce.
Summary updated