Divergence Measures Tool:An Introduction with Brief Tutorial

Douglas Briesch, Claire Jaja, Terrence J. Moore, Clare R. Voss2014 · DOI

TL;DR

This paper introduces the Divergence Measures Tool (DMTool), a software application designed to quantify the dissimilarity between text corpora. By employing seven information-theoretic divergence measures, the DMTool helps researchers understand how different text collections are, which is crucial for tasks like building natural language processing models.

What Problem Does This Paper Address?

In natural language processing (NLP), particularly for tasks like building machine translation (MT) engines or text classifiers, it's essential to understand the relationships between different text datasets. Researchers need to know how "dissimilar" two corpora are to effectively train and evaluate models. For instance, when building a classifier, corpora that are more dissimilar tend to lead to better performance. However, a standardized and accessible tool to quantify this dissimilarity across various text types was lacking. This paper addresses this by presenting the DMTool, which offers a suite of divergence measures to systematically compare text corpora.

Methods

The DMTool calculates divergence measures between pairs of text files (corpora). It supports seven information-theoretic divergence measures: Rényi, Kullback-Leibler, Bhattacharyya, Jensen-Shannon, Variational (L1), Euclidean (L2), and Cosine. The core functionality involves:

  • Uploading and Preprocessing: Users can upload text files (.txt format) and apply preprocessing steps like converting text to lowercase and tokenizing punctuation to ensure consistent term counting.
  • Divergence Calculation: The tool computes pairwise comparisons using the selected divergence measures.
  • Output and Visualization: Results are presented in various list views showing term frequencies and divergence scores. Users can also generate reports in Excel format.
  • Multiple Tabs for Use Cases: The tool is organized into tabs (Basic, Classifier, Batch, Super Batch) to support different use cases, such as classifier construction, within-corpus analysis, and assessment of line-aligned parallel files.

Datasets

The paper does not explicitly mention specific datasets used for training or evaluation beyond the included sample files ("Sample 1.txt" and "Sample 2.txt") for the tutorial. The tool is designed to work with user-provided text files.

Key Insights and Ideas

  • Quantifying Text Dissimilarity is Crucial: The paper highlights that understanding the dissimilarity between text corpora is fundamental for many NLP tasks, directly impacting model performance.
  • A Unified Tool for Divergence Measures: The DMTool consolidates multiple divergence measures into a single, user-friendly interface, making it accessible for researchers.
  • Preprocessing Matters: The tutorial demonstrates how preprocessing steps like lowercasing and punctuation tokenization significantly affect the divergence scores by ensuring consistent term identification.
  • Multiple Divergence Measures Offer Different Perspectives: The availability of seven different measures suggests that no single measure is universally best, and researchers can choose the most appropriate one for their specific needs.

Key Results

The paper focuses on introducing the tool and its functionality rather than presenting empirical results from large-scale experiments. The tutorial section demonstrates how the tool produces divergence scores. For example, in the tutorial, after applying preprocessing, the divergence scores between "Sample 1.txt" and "Sample 2.txt" are calculated, showing significant differences (e.g., Kullback-Leibler scores of 3.0313 and 2.9117). The paper also shows examples of how the tool can identify highly divergent partitions within a corpus, as illustrated in the "Batch Report" section where color-coded matrices highlight differences.

Limitations and Future Work

  • Current Version Limitations: The current version of the DMTool primarily supports ASCII or UTF-8 encoding and does not handle UTF-16 encoding. It also has limitations with generating reports in older versions of Microsoft Excel.
  • Future Work: The authors mention several functionalities under consideration for future versions, including:
  • Reading files with UTF-16 encoding.
  • Automatically generating Venn diagrams for type counts.
  • Calculating corpus perplexity.
  • Improving error handling for Excel reports.
  • Allowing personalized punctuation lists for tokenization.
  • Calculating measures and word lists by n-grams.

Summary updated

Abstract

This report provides new users of the Divergence Measures Tool (DMTool) with an overview of its core functionality. The DMTool calibrates natural language text corpora for how dissimilar they are from each other, based on the distribution of the relative frequencies of the terms in each corpus. Computation involves a suite of seven information-theoretic divergence measures calculated on given pairs of text files. Users are provided with the resulting scores and list views of the file terms with their frequencies, as used in computing the scores. Use cases and a hands-on tutorial are provided in this report.
BibTeX
@misc{briesch2014divergence,
  author = {Douglas Briesch and Claire Jaja and Terrence J. Moore and Clare R. Voss},
  title = {{Divergence Measures Tool:An Introduction with Brief Tutorial}},
  year = {2014},
  doi = {10.21236/ada599155},
}
Figure