Addressing Class Imbalance in Text Classification

172 concepts165 papers3 proposals

Text classification models thrive on balance. When a machine learning model learns to distinguish between categories, it attempts to draw a decision boundary in a high-dimensional semantic space. However, in many real-world applications, the data is heavily skewed. For instance, in document routing tasks, relevant documents are often orders of magnitude rarer than irrelevant ones, as noted by Stephen E. Robertson et al. in Microsoft Cambridge at TREC 2002: Filtering Track. Similar challenges arise when detecting rare but critical linguistic phenomena, such as offensive language (Karishma Laud et al. in problemConquero at SemEval-2020 Task 12: Transformer and Soft Label-based Approaches) or patronizing language (Marco Siino et al. in McRock at SemEval-2022 Task 4: Patronizing and Condescending Language Detection using Multi-Channel CNN, Hybrid LSTM, DistilBERT and XLNet).

When one class vastly outnumbers another, standard machine learning objectives inadvertently optimize for global accuracy by defaulting to the majority class. Geometrically, the dense cluster of majority-class vectors overwhelms the sparse minority-class vectors, causing the decision boundary to collapse inward until the minority class is effectively ignored. Understanding why this happens—and how to fix it—requires looking at feature extraction, optimization objectives, and data representation.

The Failure of Frequency-Based Representations

The problem of class imbalance begins long before a classifier makes a prediction; it is frequently embedded in how text features are extracted. Traditional text representation techniques like TF-IDF inherently favor terms that appear frequently across the dominant classes. Because the background corpus is dominated by the majority class, the unique vocabulary of minority categories is often treated as statistical noise.

To counteract this, Zhenyan Liu et al. proposed a shift from raw frequency to discriminative power in A Novel Term Selection Approach in sLDA for Imbalanced Text Categorization. By replacing the inverse document frequency (IDF) factor with Information Gain (IG)—creating a tf-IG metric—the authors ensure that terms highly predictive of rare classes are preserved rather than filtered out.

Similarly, moving away from sparse, discrete word counts toward dense, class-specific representations helps capture the shared semantic relationships within minority categories. This mechanism was explored by Peng Jin et al. in Bag-of-Embeddings for Text Classification, demonstrating that dense semantic spaces prevent rare classes from being isolated as uninformative outliers.

Reshaping the Optimization Landscape

If a loss function treats every misclassification equally, a model training on a dataset with a 99:1 imbalance can achieve 99% accuracy simply by guessing the majority class every time. To force the model to learn meaningful representations for rare classes, researchers modify the optimization landscape using advanced loss functions.

The most direct intervention is to apply class-weighted loss functions, which penalize errors on the minority class more severely than errors on the majority class. For instance, Leon Hecht et al. leveraged weighted loss functions in PCIC at SMM4H 2024: Enhancing Reddit Post Classification on Social Anxiety Using Transformer Models and Advanced Loss Functions to prevent deep transformer models from defaulting to dominant categories.

Going a step further, focal loss dynamically scales the penalty based on prediction confidence. Instead of assigning a static weight to a class, it down-weights easy-to-classify examples (which typically belong to the majority class) and forces the model to focus on hard, ambiguous examples. This strategy was utilized by Nirmal Manoj et al. in iREL at SemEval-2023 Task 10: Multi-level Training for Explainable Detection of Online Sexism to improve performance on highly skewed social media datasets, ensuring that the model's gradients are driven by the nuances of the minority class rather than the sheer volume of the majority class.

Artificially Populating Sparse Spaces

When algorithmic adjustments to the loss function are insufficient, researchers turn to data-level interventions to balance the underlying distribution directly. The goal here is to populate the sparse regions of the semantic space belonging to the minority class, either through synthetic over-sampling or structured data augmentation.

In Recommendation and sentiment classification on E-Commerce reviews, Tianyi Lin evaluated the trade-offs between synthetic over-sampling and ensemble-based under-sampling, highlighting that preserving the underlying characteristics of minority classes remains a delicate balance. If over-sampling is too simplistic (e.g., naive duplication), the model overfits; if under-sampling is too aggressive, valuable majority-class context is lost.

To generate high-quality text augmentations without relying on expensive generative models, Romain Meunier et al. introduced SRL4NLP in Image and Text: Fighting the same Battle? Super Resolution Learning for Imbalanced Text Classification. Borrowing the concept of "super-resolution" from computer vision, they mapped short, noisy text (low-resolution) to long, structured text (high-resolution) to effectively expand rare data classes.

Additionally, unsupervised text augmentation strategies, such as those explored by Daryna Dementieva et al. in SkoltechNLP at SemEval-2020 Task 11: Exploring Unsupervised Text Augmentation for Propaganda Detection, offer a way to shift the precision-recall balance by expanding the linguistic variety of underrepresented categories. By artificially densifying the minority class's footprint in the embedding space, these methods ensure that classifiers can establish robust, stable decision boundaries.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_class_imbalance_in_text_classification_b279bf0f,
  title = {class imbalance in text classification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/class-imbalance-in-text-classification/rdr_b279bf0fc1184eaebde9efb7143ffb2a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_class_imbalance_in_text_classification_b279bf0f_v1,
  title = {class imbalance in text classification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/class-imbalance-in-text-classification/rdr_b279bf0fc1184eaebde9efb7143ffb2a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake