Heavy-tailed Representations, Text Polarity Classification & Data Augmentation

Hamid Jalalzai, Pierre Colombo, Chloé Clavel, Éric Gaussier, Giovanna Varni, Emmanuel Vignon, Anne Sabourin

Introduction

Most modern Natural Language Processing (NLP) models, such as BERT, are optimized to perform well on the "bulk" of a data distribution—the common, average cases. However, human language is inherently "heavy-tailed." Following principles like Zipf's Law, a small number of words and patterns are extremely common, while the vast majority are rare.

In classification tasks, samples that fall into these distributional tails are often longer, more complex, or contain rare linguistic structures. Because these samples are scarce, standard classifiers struggle to generalize to them. Furthermore, traditional data augmentation techniques (like synonym swapping) rely on brittle heuristics that cannot guarantee the underlying meaning or "label" of the sentence remains intact.

To address this, researchers have introduced a novel framework applying Multivariate Extreme Value Theory (EVT) to NLP. By learning "heavy-tailed" text representations, this approach improves classification for rare data points and leverages a unique mathematical property—scale invariance—to generate label-consistent synthetic text.

Methodology

The researchers propose two primary contributions to handle extreme linguistic cases: Learning a Heavy-Tailed Representation (LHTR) and Generating Label Invariant sequences from Extremes (GENELIEX).

1. LHTR: Learning a Heavy-Tailed Representation

Standard embeddings do not inherently possess the regularity properties required by Extreme Value Theory. To fix this, LHTR employs an adversarial training strategy to map standard embeddings (like those from BERT) into a new latent space, $Z$.

Latent space Z showing bulk and extreme representations

This new space is forced to follow a heavy-tailed target distribution (specifically, a multivariate logistic distribution). By structuring the latent space this way, the model can utilize a specialized "angular classifier." In the tail region of a heavy-tailed distribution, the direction (angle) of a data point carries the most information about its class, while its magnitude (distance from the center) represents how "extreme" it is. This allows the classifier to focus specifically on the angular direction of vectors in the tail region.

2. GENELIEX: Generating Label Invariant Sequences

Data augmentation is often a hit-or-miss endeavor. However, in the heavy-tailed latent space created by LHTR, the optimal classifier for extreme values is scale-invariant.

Scale invariance in the extreme region

Mathematically, this means if a vector $x$ in the extreme region is labeled "positive," then any scaled version $\lambda x$ (where $\lambda > 1$) is guaranteed to also be "positive." GENELIEX exploits this mathematical "safe zone." It takes an extreme sample, scales its latent representation, and then uses a Transformer decoder to translate that scaled vector back into a new synthetic sentence. Because of the scale-invariance property, the newly generated sentence is mathematically incentivized to retain the original sentiment label.

Experimental Results

The authors evaluated their methods on two massive sentiment analysis datasets: Amazon Reviews (231,000 samples) and Yelp Reviews (1.45 million samples). Ratings of 4 or 5 were labeled positive (+1), while 1 or 2 were labeled negative (-1). They also tested smaller subsets (1k and 10k samples) to evaluate augmentation performance in data-scarce environments.

Improved Classification on Hard Samples

The authors found that "extreme" samples in their latent space tended to be longer and more difficult for standard models to handle, confirming EVT as a valid tool for isolating hard linguistic cases. The best results were achieved using a hybrid "Proposed Model" that uses a standard neural network for the bulk of the data and the LHTR angular classifier specifically for the 25% most extreme samples.

On the Yelp dataset, this hybrid model achieved a classification loss of 0.103, significantly outperforming a standard BERT-based neural network (0.111). The improvement was most pronounced in the extreme regions (0.1205 vs. 0.148).

Superior Data Augmentation

GENELIEX outperformed state-of-the-art augmentation methods, including Easy Data Augmentation (EDA) and contextual augmentation. On the Amazon "Medium" dataset, GENELIEX reached an F1 score of 86.3, compared to 85.2 for the next best method. Furthermore, GENELIEX produced significantly more diverse text (measured by unigram and bigram diversity) than its competitors, helping to prevent models from overfitting on repetitive synthetic data.

Limitations and Future Work

While highly effective, the framework has a few limitations:

  • Extreme-Only Augmentation: Currently, GENELIEX only works for samples that fall into the "extreme" regions of the distribution. The scale-invariance trick cannot yet be applied to augment the "bulk" of the data.
  • Computational Overhead: The method requires training an adversarial encoder and a specialized decoder on top of existing models like BERT, which adds architectural complexity.

Despite these limitations, the introduction of Extreme Value Theory to NLP opens exciting new doors. The authors suggest that heavy-tailed representations could eventually be applied to other modalities—such as audio, video, or medical records—where accurately modeling "extreme" anomalies or rare events is of critical importance.

Summary updated

Abstract

The dominant approaches to text representation in natural language rely on learning embeddings on massive corpora which have convenient properties such as compositionality and distance preservation. In this paper, we develop a novel method to learn a heavy-tailed embedding with desirable regularity properties regarding the distributional tails, which allows to analyze the points far away from the distribution bulk using the framework of multivariate extreme value theory. In particular, a classifier dedicated to the tails of the proposed embedding is obtained which performance outperforms the baseline. This classifier exhibits a scale invariance property which we leverage by introducing a novel text generation method for label preserving dataset augmentation. Numerical experiments on synthetic and real text data demonstrate the relevance of the proposed framework and confirm that this method generates meaningful sentences with controllable attribute, e.g. positive or negative sentiment.
BibTeX
@inproceedings{jalalzai2020heavy,
  author = {Hamid Jalalzai and Pierre Colombo and Chloé Clavel and Éric Gaussier and Giovanna Varni and Emmanuel Vignon and Anne Sabourin},
  title = {{Heavy-tailed Representations, Text Polarity Classification \& Data Augmentation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=GquPYhfqEw},
}
Figure