Embarrassingly Simple Performance Prediction for Abductive Natural Language Inference

Emils Kadikis, Vaibhav Srivastav, Roman KlingerNAACL 2022 · DOI

Abstract

Selecting the optimal pre-trained transformer model for complex reasoning tasks often requires computationally expensive fine-tuning. This paper introduces a "fast-track" method to predict model performance on Abductive Natural Language Inference ($\alpha$NLI) by measuring the geometric similarity between sentence embeddings. This approach ranks model performance 620 times faster than traditional fine-tuning while maintaining a strong correlation with final classification results, offering a sustainable and efficient alternative for model selection.

Introduction

In the current "Golden Age" of Natural Language Processing (NLP), researchers have access to tens of thousands of pre-trained transformer models via platforms like HuggingFace. However, identifying the most effective model for a specific task remains a significant bottleneck. The standard approach—fine-tuning multiple candidate models on a specific dataset—is both computationally and environmentally costly.

This research focuses on Abductive Natural Language Inference ($\alpha$NLI), a challenging task requiring common-sense reasoning. In $\alpha$NLI, a model must select the most plausible explanation for a given set of observations. Given the complexity of the task, efficient model selection is critical.

Methodology: The Similarity Proxy

The authors propose an "embarrassingly simple" proxy for performance based on Cosine Similarity. The process involves three main steps:

  1. Sentence Embeddings: Pre-trained models are used to generate mathematical vectors (embeddings) for sentences via "mean pooling" before any task-specific training occurs.
  2. The Similarity Approach: To solve an $\alpha$NLI problem, observations ($o_1, o_2$) and hypothesis choices ($h_1, h_2$) are represented as vectors. The model predicts the hypothesis whose vector is geometrically closest (highest cosine similarity) to the observation vector.
  3. Correlation Analysis: The researchers tested 17 different transformer architectures to determine if models performing well under this simple similarity metric also excelled after full classification training.

Experimental Setup

The study utilized the ART (Abductive Reasoning Test) dataset, which consists of 169,654 training samples and 1,532 validation samples. Each sample contains two observations and two manually curated hypotheses. The validation set employs "adversarial filtering" to eliminate statistical shortcuts, ensuring models rely on genuine reasoning.

Key Results and Discussion

The experimental results demonstrate that the internal latent space of a pre-trained model serves as a powerful predictor of its eventual fine-tuned performance.

Performance and runtime comparison across models

As shown in the data above, the similarity-based method provides a massive speedup—averaging 620 times faster than full fine-tuning. While the absolute accuracy of the similarity method is relatively low (hovering around 51-52%), its ability to rank models is highly effective.

Correlation between similarity and classification accuracy

Statistical analysis revealed a strong relationship between the proxy and the final results, with a Pearson coefficient of 0.65 and a Spearman rank correlation of 0.67. Models like ELECTRA-large emerged as leaders in both metrics, while others like MobileBERT consistently lagged behind. This suggests that if a model's latent space already clusters logically related concepts, it is inherently better suited for the downstream task.

Conclusion and Future Work

While the similarity method is a robust predictor of performance, it is not a replacement for training; fine-tuned models still achieve significantly higher absolute accuracy (up to 88%). Furthermore, this study focused exclusively on $\alpha$NLI. Future research is required to determine if this "embarrassingly simple" technique generalizes to other NLP tasks such as summarization or question answering. Nonetheless, the method provides a vital tool for reducing the environmental and computational footprint of modern AI research.

Summary updated

BibTeX
@inproceedings{kadikis2022embarrassingly,
  author = {Emils Kadikis and Vaibhav Srivastav and Roman Klinger},
  title = {{Embarrassingly Simple Performance Prediction for Abductive Natural Language Inference}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-main.441},
}
Figure