Proceedings of the Second Workshop on Insights from Negative Results in NLP

João Sedoc, Anna Rogers, Anna Rumshisky, Shabnam Tafreshi, Abeer Aldayel, Amittai Axelrod, Mark Anderson, Sawsan Alqahtani, Wazir Ali, Aditya Bhargava, Torento, Federico Bianchi, Jeremy Barnes, Vasudha Bhatnagar, Yash Butala, Meghana Bhat, Shubham Chatterjee, Somnath Basu, Roy Chowdhury, Aleksandr Drozd, Alexandra Delucia, Luis Fernando, D Haro, Darren Edmonds, Salvatore Giorgi, Pedram Hosseini, Sardar Hamidian, Christopher Klamm, Huda Khayrallah, Neha Kennard, Alka Khurana, Yiyuan Li, Andrei Romascanu, Ansel Maclaughlin, Ashutosh Modi, Deepa Muralidhar, Markus Mller, Tristan Naumann, Jessica Ouyang, John Ortega, J Morgen, Adam Poliak, Ellie Pavlick, Paola Perera, Michal Ptaszynski, Neville Ryant, Ali Seyfi, Michael Saxon, Mihai Surdeanu, Sashank Santhanam, Vivek Srivastava, Ting Shiue, Silvia Terragni, Derry Wijaya, Jin-Ge Yao, Xiang Zhou, Zachary Lipton, Rachael Tatman, Bonnie Rasa, Webber, Hinrich Schtze, Yuxian Zhu, Lingxiao Gu, Bing Luo, L Cheng, Wei Peng, Minlie Huang, Xiaoyan Zhu, ; Rosendahl, Christian Herold, Frithjof Petrick, Hermann Ney, Daphn Chopard, Matthias, Liane Guillou, Sander Bijl De Vroe, Mark Johnson, Mark Steedman, Nataliia Kees, Michael Fromm, Evgeniy Faerman, Thomas SeidlOpenalex Unknown 2021 · DOI · OpenAlex

The pursuit of State-of-the-Art (SOTA) performance has long dominated Natural Language Processing (NLP) research. However, the Proceedings of the Second Workshop on Insights from Negative Results in NLP (ACL 2021) suggests that what doesn't work is often as important as what does. By providing a venue for well-executed experiments that yielded null results or failed to beat benchmarks, this collection offers a necessary reality check on the current trajectory of the field.

The Problem of Publication Bias

In the current NLP landscape, there is an overwhelming pressure to publish only "positive" results—new architectures or training methods that achieve record-breaking performance. The workshop organizers argue that this creates a publication bias that turns NLP development into a cycle of "tinkering and tweaking" rather than rigorous science.

When researchers do not share their failures, the community wastes immense time, computational power, and carbon emissions reinventing the wheel or pursuing dead-end hypotheses. This workshop aims to bridge that gap by providing a venue for "failed" ideas that nonetheless offer deep scientific insight into model behavior.

A Taxonomy of Negative Results

The proceedings categorized contributions into several critical areas of inquiry:

  • Ablation Studies: Identifying components that, despite being considered essential, contribute little to final performance.
  • Generalization Failures: Highlighting the fragility of models when moved beyond their training distributions or into new languages.
  • Stability Testing: Documenting how sensitive results are to random seeds, hardware variations, or minor preprocessing changes.
  • Trivial Baselines: Showing that complex deep learning architectures sometimes barely outperform simple, classical algorithms.
  • Theoretical Proofs: Establishing mathematical boundaries for why certain approaches are fundamentally unsuited for specific tasks.

Key Insights and Debunked Assumptions

The papers within these proceedings utilize a broad spectrum of standard benchmarks—including Natural Language Inference (NLI), Neural Machine Translation (NMT), and Task-Oriented Dialog—to challenge common industry assumptions:

The "More is Better" Fallacy

Several papers suggest that increasing model complexity, such as adding extra attention heads or extending Masked Language Model (MLM) pre-training, often yields diminishing or even zero returns for specific tasks like dialog or morphological inflection.

Heuristic Over-reliance

A key insight in the NLI research is that models are often not "learning" language in a human-like sense; instead, they are becoming highly proficient at identifying statistical shortcuts and heuristics within the dataset.

The Power of the Baseline

One notable paper highlights that a "Corrected CBOW" (a simpler, older architecture) can perform just as well as the more popular Skip-gram model. This suggests that the field may have moved away from simpler, more efficient tools too quickly in favor of complexity.

BERT’s Blind Spots

Research in this collection shows that BERT struggles significantly with character-level alignment and is surprisingly sensitive to "native interference" when processing L2 (second language) text, revealing limitations in its architectural universality.

Quantitative Overview

The workshop saw a significant uptick in interest compared to previous years, indicating a potential shift in the field's culture:

  • Submissions: 39 papers were submitted, with 20 short papers accepted (a competitive 51.2% acceptance rate).
  • Top Themes: The majority of papers (19) focused on "great ideas that didn't work," followed by 11 papers on generalizability issues.
  • Performance Gains: Quantitative results across the papers often showed that "standard" improvements in NMT or Variational Autoencoder (VAE) frameworks provided 0% to negligible gains when tested under more rigorous or "fair" conditions.

The Path Forward

The organizers acknowledge that fighting the "positive result" bias is an uphill battle. Historically, similar efforts have struggled to maintain momentum due to a lack of submissions or perceived prestige. To ensure the long-term health of the field, the workshop identifies three future priorities:

  1. Encouraging Reproducibility: Prioritizing papers that attempt to replicate famous results and document the failures encountered.
  2. Developing New Metrics: Creating standards for what makes a "good" negative result, such as how much "search space" it clears for other researchers.
  3. Mainstream Integration: Moving these reality checks into the main tracks of major conferences rather than keeping them confined to specialized workshops.

By documenting the paths that lead nowhere, the NLP community can more efficiently find the paths that lead forward.

Summary updated

BibTeX
@inproceedings{sedoc2021proceedings,
  author = {João Sedoc and Anna Rogers and Anna Rumshisky and Shabnam Tafreshi and Abeer Aldayel and Amittai Axelrod and Mark Anderson and Sawsan Alqahtani and Wazir Ali and Aditya Bhargava and Torento and Federico Bianchi and Jeremy Barnes and Vasudha Bhatnagar and Yash Butala and Meghana Bhat and Shubham Chatterjee and Somnath Basu and Roy Chowdhury and Aleksandr Drozd and Alexandra Delucia and Luis Fernando and D Haro and Darren Edmonds and Salvatore Giorgi and Pedram Hosseini and Sardar Hamidian and Christopher Klamm and Huda Khayrallah and Neha Kennard and Alka Khurana and Yiyuan Li and Andrei Romascanu and Ansel Maclaughlin and Ashutosh Modi and Deepa Muralidhar and Markus Mller and Tristan Naumann and Jessica Ouyang and John Ortega and J Morgen and Adam Poliak and Ellie Pavlick and Paola Perera and Michal Ptaszynski and Neville Ryant and Ali Seyfi and Michael Saxon and Mihai Surdeanu and Sashank Santhanam and Vivek Srivastava and Ting Shiue and Silvia Terragni and Derry Wijaya and Jin-Ge Yao and Xiang Zhou and Zachary Lipton and Rachael Tatman and Bonnie Rasa and Webber and Hinrich Schtze and Yuxian Zhu and Lingxiao Gu and Bing Luo and Li and L Cheng and Wei Peng and Minlie Huang and Xiaoyan Zhu and ; Rosendahl and Christian Herold and Frithjof Petrick and Hermann Ney and Daphn Chopard and Matthias and Liane Guillou and Sander Bijl De Vroe and Mark Johnson and Mark Steedman and Nataliia Kees and Michael Fromm and Evgeniy Faerman and Thomas Seidl},
  title = {{Proceedings of the Second Workshop on Insights from Negative Results in NLP}},
  booktitle = {Openalex Unknown},
  year = {2021},
  doi = {10.18653/v1/2021.insights-1},
}
Figure