Introduction
Deep learning models often prioritize the "path of least resistance" during training. In Natural Language Inference (NLI), this manifests as a reliance on shallow heuristics—simple syntactic shortcuts that allow a model to achieve high benchmark scores without developing a genuine semantic understanding of the text. This paper investigates whether Focal Loss, a technique originally designed for addressing class imbalance in computer vision, can be repurposed to force models away from these shortcuts and toward more robust reasoning.
The Problem: Shallow Heuristics in NLI
In NLI, models must determine if a premise entails a hypothesis. However, standard datasets like SNLI and MNLI often contain unintended biases. For example, if a hypothesis significantly overlaps with its premise, it is statistically more likely to be an "entailment." Models quickly learn these patterns as shortcuts.
When these models are tested on adversarial datasets like HANS (Heuristic Analysis for NLI Systems), which contains counter-examples to these heuristics, their performance often collapses to near-zero accuracy on non-entailed samples. The authors treat these counter-examples as a "minority class" that the model ignores in favor of the easier majority patterns.
Methodology: Modulating Cross-Entropy
The researchers propose replacing standard Cross-Entropy (CE) loss with Focal Loss (FL). The core idea is to treat heuristic-based "easy" samples as a majority class that should be down-weighted once the model has learned them, thereby focusing the "gradient budget" on harder, more informative examples.
The Focal Loss formula introduces a modulating factor $(1 - p_t)^\gamma$: $$FL(p_t) = -(1 - p_t)^\gamma \log(p_t)$$
As the parameter $\gamma$ increases, the loss for correctly classified samples (where $p_t$ is high) drops significantly. This prevents the model from becoming overconfident on easy shortcuts and encourages it to resolve more complex linguistic cases. The authors tested this using both BERT (transformer) and InferSent (BiLSTM) architectures.
Experimental Results
Out-of-Distribution (OOD) Generalization
On the HANS dataset, BERT trained with Focal Loss ($\gamma=5$) saw a notable improvement, with accuracy rising from 52.20% (Cross-Entropy) to 59.54%. This suggests that the loss modulation successfully discouraged some reliance on the lexical overlap and constituent heuristics.
The In-Distribution Trade-off
The gains in OOD robustness came at a cost to in-distribution performance. On the MNLI test set, accuracy fell from 83.82% (CE) to 82.65% ($\gamma=5$). Furthermore, Focal Loss surprisingly decreased performance on the "HARD" subsets of MNLI and SNLI. This indicates that while FL helps avoid specific syntactic shortcuts, it does not necessarily equip the model with the complex reasoning required for genuinely difficult linguistic tasks.
Discussion and Limitations
Focal Loss acts as a "blind" regularizer. While it identifies which samples are difficult based on the model's current predictions, it cannot distinguish between a sample that is hard because it challenges a heuristic and one that is hard because it is noisy or mislabeled.
Moreover, the Validation Paradox remains a significant hurdle: the benefits of Focal Loss are most apparent on OOD data, but hyperparameter tuning (selecting $\gamma$) typically relies on in-distribution validation sets. Without access to adversarial data during the tuning phase, finding the optimal $\gamma$ is challenging. Plain Focal Loss also remains less effective than specialized debiasing methods, such as those using auxiliary models to explicitly identify biases.
Conclusion
Focal Loss provides a simple, model-agnostic way to improve the robustness of NLI systems against shallow heuristics. However, it introduces a trade-off in standard benchmark accuracy and is less effective than targeted debiasing ensembles. Future work should explore more expressive models and better validation strategies to harness the full potential of modulated loss functions.
Summary updated