Proceedings of the Fourteenth Workshop on Semantic Evaluation

Aurélie Herbelot, Nathan Schneider, Jonathan May, Eneko Agirre, Timothy Baldwin, Miguel Ballesteros, Francesco Barbieri, Valerio Basile, Kalina Bontcheva, Georgeta Bordea, Davide Buscaldi, José Camacho-Collados, Marine Carpuat, Daniel Cer, Thierry Charnois, Wanxiang Che, Colin Cherry, Mrinal K. Das, Tobias Daudert, Mona Diab, Luis Espinosa Anke, André Freitas, Dimitris Galanis, Iryna Gurevych, Ivan Habernal, Chris- Tian Hempelmann, Véronique Hoste, Filip Ilievski, David Jurgens, Svetlana Kiritchenko, Alicia Krebs, Egoitz Laparra, Els Lefever, Alessandro Lenci, Maria Liakata, Iñigo López-Gazpio, Wei Lü, Montse Maritxalar, Tristan Miller, Ashutosh Modi, Saif M. Mohammad, Alessandro Moschitti, Hamdy Mubarak, Preslav Nakov, Roberto Navigli, Sergio Oramas, Simon Ostermann, Gustavo Henrique Paetzold, Denis Paperno, Tommaso Pasini, Viviana Patti, Mohammad Taher Pilehvar, Maria Pontiki, Marten Postma, Peter Potash, James Pustejovsky, Behrang QasemiZadeh, Alan Ritter, Alexey Romanov, Francesco Ronzano, Sara Rosenthal, Anna Rumshisky, Horacio Sag- Gion, Mohammad Salameh, Juliano Efson Sales, Anne-Kathrin Schumann, Piek Vossen, Henning Wachsmuth, Haïfa Zargayouna, Manel Zarrouk, Arkaitz Zubiaga, Just ; Hr, Al-Smadi Mohammad, Kk, Kristian Nrgaard Jensen, Nicolaj Filrup Rasmussen, Thai Wang, Marco Placenti, Barbara, Nuaa-Qmul At Semeval, Prhlt-Upv At Semeval, Elmo-Nb At Semeval, Semeval Mlengineer, Gaku Hitachi At Semeval ; Morishita, Shota Morio, Hiroaki Horiguchi, Toshinori Ozaki, Miyoshi, Mayukh Sharma, Ilanthenral Kandasamy, W Vasantha, Ayrton Amaral, Ndia Silva, Anderson Da, Silva Soares, Team_Swift At, Marcos ZampieriOpenalex Unknown 2020 · DOI · OpenAlex

Introduction

The Fourteenth Workshop on Semantic Evaluation (SemEval-2020) represents a landmark collection of research in the field of Natural Language Processing (NLP). Comprising 300 papers—including 12 task overviews and 288 system descriptions—these proceedings document the collective efforts of the global research community to push the boundaries of machine language understanding. SemEval remains a critical venue for evaluating state-of-the-art models on diverse, challenging semantic tasks that move beyond basic linguistic processing into the realm of deep reasoning.

The Core Challenges: Four Research Tracks

The 2020 workshop organized its 12 shared tasks into four primary tracks, reflecting the evolving priorities of the NLP community:

  1. Lexical Semantics: This track focused on the fluid nature of language, including tasks to detect diachronic word-meaning shifts (how meanings change over centuries) and cross-lingual alignment.
  2. Common Sense Knowledge and Reasoning: Moving beyond simple pattern matching, these tasks challenged models to validate "common sense" statements and provide explanations for why certain sentences are nonsensical.
  3. Humor, Emphasis, and Sentiment: Researchers explored the nuances of human expression by quantifying the "funniness" of edited headlines and analyzing the complex emotional layers of internet memes.
  4. Societal Applications of NLP: Addressing real-world harms, this track included tasks for detecting propaganda techniques in news and identifying offensive language across multilingual social media environments.

Technical Landscape: The Transformer Era

The technical contributions within the 288 system papers reveal a clear consensus on methodology. The dominance of Transformer-based architectures is absolute, with models such as BERT, RoBERTa, ALBERT, and XLM-R serving as the foundational architecture for nearly every competitive entry.

Beyond the choice of base model, several key technical trends emerged across the submissions:

  • Ensemble Engineering: Top-performing systems frequently utilized "stacking" or complex ensembles to combine the strengths of different architectures and mitigate individual model weaknesses.
  • Data Augmentation: To address data scarcity in specialized tasks like propaganda detection, teams employed creative techniques such as back-translation and pseudo-labeling to expand their training sets.
  • Multilingual Transfer: With tasks like OffensEval 2020 spanning five languages, the use of cross-lingual embeddings and zero-shot transfer learning became a standard requirement for robust performance.

Dataset Innovation and Multimodality

SemEval-2020 acted as a "dataset factory," introducing several high-impact benchmarks that continue to influence the field. Notable among these were:

  • Task 4 (ComVE): A dataset specifically designed for Commonsense Validation and Explanation.
  • Task 7 (Humicroedit): An exploration of computational humor through news headlines edited for comedic effect.
  • Task 8 (Memotion): A first-of-its-kind multimodal dataset for meme emotion analysis, which highlighted the difficulty of reconciling visual and textual cues.
  • Task 11 & 12: Specialized datasets addressing propaganda detection (identifying 18 distinct techniques) and multilingual offensive language identification.

Key Findings and the "Reasoning Gap"

The results of the workshop highlight both the progress and the persistent limitations of current NLP. While models have reached high accuracy—sometimes exceeding 90%—in tasks like offensive language detection, a significant "reasoning gap" remains. Models excel at identifying surface-level patterns but struggle with tasks requiring deep world knowledge or nuanced reasoning.

Furthermore, the "Memotion" task demonstrated that multimodality is a significant frontier. The meaning of a meme often resides in the ironic tension between an image and its text; current models struggle to capture this emergent meaning compared to processing single modalities in isolation. This suggests that future research must focus on better integration of visual and linguistic features.

Conclusion and Future Directions

Despite the challenges of a global pandemic necessitating a transition to a fully virtual format, SemEval-2020 saw record participation and the introduction of new awards to incentivize creative task design and rigorous error analysis. The proceedings suggest a shift in the community toward addressing societal impact and "fake news" while identifying the need for more interpretable and robust models.

Future work suggested by the workshop organizers includes expanding language coverage to low-resource languages and moving beyond "black box" architectures toward systems that can explain their semantic decisions, particularly in sensitive areas like propaganda and sentiment analysis.

Summary updated

BibTeX
@inproceedings{herbelot2020proceedings,
  author = {Aurélie Herbelot and Nathan Schneider and Jonathan May and Eneko Agirre and Timothy Baldwin and Miguel Ballesteros and Francesco Barbieri and Valerio Basile and Kalina Bontcheva and Georgeta Bordea and Davide Buscaldi and José Camacho-Collados and Marine Carpuat and Daniel Cer and Thierry Charnois and Wanxiang Che and Colin Cherry and Mrinal K. Das and Tobias Daudert and Mona Diab and Luis Espinosa Anke and André Freitas and Dimitris Galanis and Iryna Gurevych and Ivan Habernal and Chris- Tian Hempelmann and Véronique Hoste and Filip Ilievski and David Jurgens and Svetlana Kiritchenko and Alicia Krebs and Egoitz Laparra and Els Lefever and Alessandro Lenci and Maria Liakata and Iñigo López-Gazpio and Wei Lü and Montse Maritxalar and Tristan Miller and Ashutosh Modi and Saif M. Mohammad and Alessandro Moschitti and Hamdy Mubarak and Preslav Nakov and Roberto Navigli and Sergio Oramas and Simon Ostermann and Gustavo Henrique Paetzold and Denis Paperno and Tommaso Pasini and Viviana Patti and Mohammad Taher Pilehvar and Maria Pontiki and Marten Postma and Peter Potash and James Pustejovsky and Behrang QasemiZadeh and Alan Ritter and Alexey Romanov and Francesco Ronzano and Sara Rosenthal and Anna Rumshisky and Horacio Sag- Gion and Mohammad Salameh and Juliano Efson Sales and Anne-Kathrin Schumann and Piek Vossen and Henning Wachsmuth and Haïfa Zargayouna and Manel Zarrouk and Arkaitz Zubiaga and Just ; Hr and Al-Smadi Mohammad and Kk and Kristian Nrgaard Jensen and Nicolaj Filrup Rasmussen and Thai Wang and Marco Placenti and Barbara and Nuaa-Qmul At Semeval and Prhlt-Upv At Semeval and Elmo-Nb At Semeval and Semeval Mlengineer and Gaku Hitachi At Semeval ; Morishita and Shota Morio and Hiroaki Horiguchi and Toshinori Ozaki and Miyoshi and Mayukh Sharma and Ilanthenral Kandasamy and W Vasantha and Ayrton Amaral and Ndia Silva and Anderson Da and Silva Soares and Team\_Swift At and Semeval and Marcos Zampieri and Preslav Nakov and Sara Rosenthal},
  title = {{Proceedings of the Fourteenth Workshop on Semantic Evaluation}},
  booktitle = {Openalex Unknown},
  year = {2020},
  doi = {10.18653/v1/2020.semeval-1},
}
Figure