Using Focal Loss to Fight Shallow Heuristics: An Empirical Analysis of Modulated Cross-Entropy in Natural Language Inference

Frano Rajic, Ivan Stresec, Axel Marmet, Tim PostuvanarXiv 2022 · DOI · arXiv

Introduction

Deep learning models often prioritize the "path of least resistance" during training. In Natural Language Inference (NLI), this manifests as a reliance on shallow heuristics—simple syntactic shortcuts that allow a model to achieve high benchmark scores without developing a genuine semantic understanding of the text. This paper investigates whether Focal Loss, a technique originally designed for addressing class imbalance in computer vision, can be repurposed to force models away from these shortcuts and toward more robust reasoning.

The Problem: Shallow Heuristics in NLI

In NLI, models must determine if a premise entails a hypothesis. However, standard datasets like SNLI and MNLI often contain unintended biases. For example, if a hypothesis significantly overlaps with its premise, it is statistically more likely to be an "entailment." Models quickly learn these patterns as shortcuts.

Common shallow heuristics used by NLI models to "cheat" on standard benchmarks.

When these models are tested on adversarial datasets like HANS (Heuristic Analysis for NLI Systems), which contains counter-examples to these heuristics, their performance often collapses to near-zero accuracy on non-entailed samples. The authors treat these counter-examples as a "minority class" that the model ignores in favor of the easier majority patterns.

Methodology: Modulating Cross-Entropy

The researchers propose replacing standard Cross-Entropy (CE) loss with Focal Loss (FL). The core idea is to treat heuristic-based "easy" samples as a majority class that should be down-weighted once the model has learned them, thereby focusing the "gradient budget" on harder, more informative examples.

The Focal Loss formula introduces a modulating factor $(1 - p_t)^\gamma$: $$FL(p_t) = -(1 - p_t)^\gamma \log(p_t)$$

Focal Loss curves showing how increasing the focusing parameter $\gamma$ reduces the loss contribution from easy (high-probability) samples.

As the parameter $\gamma$ increases, the loss for correctly classified samples (where $p_t$ is high) drops significantly. This prevents the model from becoming overconfident on easy shortcuts and encourages it to resolve more complex linguistic cases. The authors tested this using both BERT (transformer) and InferSent (BiLSTM) architectures.

Experimental Results

Out-of-Distribution (OOD) Generalization

On the HANS dataset, BERT trained with Focal Loss ($\gamma=5$) saw a notable improvement, with accuracy rising from 52.20% (Cross-Entropy) to 59.54%. This suggests that the loss modulation successfully discouraged some reliance on the lexical overlap and constituent heuristics.

The In-Distribution Trade-off

The gains in OOD robustness came at a cost to in-distribution performance. On the MNLI test set, accuracy fell from 83.82% (CE) to 82.65% ($\gamma=5$). Furthermore, Focal Loss surprisingly decreased performance on the "HARD" subsets of MNLI and SNLI. This indicates that while FL helps avoid specific syntactic shortcuts, it does not necessarily equip the model with the complex reasoning required for genuinely difficult linguistic tasks.

Discussion and Limitations

Focal Loss acts as a "blind" regularizer. While it identifies which samples are difficult based on the model's current predictions, it cannot distinguish between a sample that is hard because it challenges a heuristic and one that is hard because it is noisy or mislabeled.

Moreover, the Validation Paradox remains a significant hurdle: the benefits of Focal Loss are most apparent on OOD data, but hyperparameter tuning (selecting $\gamma$) typically relies on in-distribution validation sets. Without access to adversarial data during the tuning phase, finding the optimal $\gamma$ is challenging. Plain Focal Loss also remains less effective than specialized debiasing methods, such as those using auxiliary models to explicitly identify biases.

Conclusion

Focal Loss provides a simple, model-agnostic way to improve the robustness of NLI systems against shallow heuristics. However, it introduces a trade-off in standard benchmark accuracy and is less effective than targeted debiasing ensembles. Future work should explore more expressive models and better validation strategies to harness the full potential of modulated loss functions.

Summary updated

Abstract

There is no such thing as a perfect dataset. In some datasets, deep neural networks discover underlying heuristics that allow them to take shortcuts in the learning process, resulting in poor generalization capability. Instead of using standard cross-entropy, we explore whether a modulated version of cross-entropy called focal loss can constrain the model so as not to use heuristics and improve generalization performance. Our experiments in natural language inference show that focal loss has a regularizing impact on the learning process, increasing accuracy on out-of-distribution data, but slightly decreasing performance on in-distribution data. Despite the improved out-of-distribution performance, we demonstrate the shortcomings of focal loss and its inferiority in comparison to the performance of methods such as unbiased focal loss and self-debiasing ensembles.
BibTeX
@misc{rajic2022using,
  author = {Frano Rajic and Ivan Stresec and Axel Marmet and Tim Postuvan},
  title = {{Using Focal Loss to Fight Shallow Heuristics: An Empirical Analysis of Modulated Cross-Entropy in Natural Language Inference}},
  year = {2022},
  eprint = {2211.13331},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.13331},
}
Figure