Abstract
In the rapidly evolving field of Artificial Intelligence (AI), the narrative often fluctuates between extreme optimism and deep skepticism. This paper investigates the historical trends of "stance" in AI research—defined as the author's relationship to prior work—using a large-scale computational approach. By analyzing over 41,000 papers from the last 35 years using a fine-tuned SciBERT model, we find that while the field has become more positive overall, a significant recent surge in critical "negative" papers has occurred since 2015. Notably, these critical works, though rare, are disproportionately influential, garnering higher citations and acceptance rates at prestigious venues.
Introduction
Traditional scientometrics often relies on citation counts or sentiment analysis of individual sentences. However, these metrics fail to capture the holistic "stance" of a research paper: whether it positions itself as an incremental improvement (Positive Stance) or as a fundamental critique of existing methodologies, evaluations, or biases (Negative Stance). Understanding these shifts is crucial for determining if AI is in a phase of rapid discovery or self-correction.
Methodology
The authors framed stance detection as a continuous regression task, mapping papers to a score between -1 (highly critical) and +1 (highly constructive).
Model and Task
A SciBERT model was fine-tuned with a linear activation layer to process paper titles and abstracts. Unlike simple sentiment analysis, this task identifies the author's positioning relative to the field's foundations.
Data and Annotation
The study utilized two primary corpora: an NLP dataset (~23,800 papers from the ACL Anthology) and an ML dataset (~18,300 papers from venues like NeurIPS and ICML). To train the model, a gold standard of 1,550 manually labeled abstracts was created. Due to the scarcity of negative papers, keyword-based heuristics were used to oversample critical works to ensure the model could effectively identify negativity.
The inter-annotator agreement (Figure 4) demonstrates robust correlation (Pearson r) and moderate-to-high Cohen's kappa (κ) across different annotator pairs, validating the reliability of the training data used for the SciBERT model.
Results and Analysis
The longitudinal analysis reveals several key trends in the "vibe" of AI research.
The Evolution of Stance
Since the mid-1980s, the average stance has trended toward positivity, reflecting a shift toward statistical grounding and cumulative progress. However, a turning point was observed around 2015. In NLP specifically, positivity peaked during the early Deep Learning revolution and has since declined as researchers increasingly question robustness, ethics, and evaluation metrics.
As shown in Figure 2, the distribution of stance scores is heavily skewed toward the positive end (+1.0), indicating that most research is constructive. However, there are distinct clusters of neutral (0.0) and highly critical (-1.0) papers. NLP papers tend to be slightly more critical on average than general ML papers.
The Negativity Paradox
While highly negative papers (stance < -0.3) account for less than 4% of the total volume, they are significantly more "disruptive." These papers receive more citations on average than their positive counterparts. Furthermore, at top-tier venues like ICLR, papers with extreme negative stances (< -0.6) exhibit higher acceptance rates, suggesting that the community highly values rigorous critiques of the status quo.
Discussion and Limitations
The study highlights a growing "extremism" in the field: positive papers are reporting results with higher confidence, while negative papers are becoming more scathing.
Several limitations remain. Historical trends prior to the 1990s are less certain due to OCR quality issues in older archives. Additionally, the reliance on abstracts may overlook nuanced critiques present only in full-text discussions. Future research should explore whether the rise in negativity is a response to a "reproducibility crisis" or a sign of the field's maturation.
Conclusion
AI research is not just about beating benchmarks; it is increasingly about critical self-reflection. For researchers seeking impact, a well-reasoned critique of existing paradigms may prove more influential than marginal incremental gains.
Summary updated