Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Tutorials

Andrew Yates, Rodrigo Nogueira, Jimmy Lin, T. B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric J. Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, J. L. Clark, Christopher Berner, Matthew Henderson, Rami Al‐Rfou, Brian Strope, You Kyung Sung, Hassan Sajjad, Narine Kokhlikyan, Fahim Dalvi, Nadir Durrani, Lingfei Wu, Jennifer cohere.com Jennifer cohere.com, Yu Facebook, Heng Ji 0001, Yunyao Li, Paul School, Daria Baidakova, Valentina Fedorova, Heng Ji, Mitesh M. Khapra, Ananya B. Sai, Fahim, Drutsa, Alexey, Hajishirzi, Ji Heng, Megorskaya, Olga, Peters, Hassan, Ustalov, Dmitry, Lingfei, Andrew, Luca Soldaini, Alessandro Moschitti, Ledell Wu, Adam Fisch, Sumit Chopra, Keith Adams, Antoine Bordes, Jason Weston, Zhijing Wu, Jiaxin Mao, Yiqun Liu, Jingtao Zhan, Yukun Zheng, Min Zhang, Shaoping Ma, Hamed Zamani, Mostafa Dehghani, W Croft, Erik Learned-Miller, Jaap Kamps, Zhi Zheng, Kai Hui, Ben He, Xianpei Han, Le Sun, Anthony Bau, Yonatan BelinkovOpenalex Unknown 2021 · DOI · OpenAlex

Abstract

The goal of text ranking is to generate an ordered list of texts retrieved from a corpus in response to a query for a particular task. Although the most common formulation of text ranking is search, instances of the task can also be found in many text processing applications. This tutorial volume provides an overview of text ranking with neural network architectures known as transformers, of which BERT (Bidirectional Encoder Representations from Transformers) is the best-known example. These models produce high-quality results across many domains, tasks, and settings.

Introduction

The collected proceedings of the tutorial sessions at the NAACL-HLT 2021 conference serve as a curated "state-of-the-union" for the most pressing frontiers in Natural Language Processing (NLP). As models like BERT and GPT have become industry standards, the research community has pivoted toward addressing the "scaling walls" associated with efficiency, interpretability, and evaluation. This paper synthesizes the strategic roadmaps provided by these tutorials for researchers and engineers working at the cutting edge of language technology.

Overcoming the Scaling Wall

Standard transformer architectures, while powerful, are computationally expensive and struggle with long sequences (typically beyond 512 tokens) or high-speed search tasks. The tutorials outline two primary methods for bypassing these bottlenecks:

  1. Neural Text Ranking: The field is moving from simple re-ranking to "learned dense representations." This allows for direct, high-speed semantic search across massive corpora, significantly outperforming traditional sparse signals like BM25.
  2. Long-Sequence Processing: To address the $O(N^2)$ memory bottleneck of standard attention, new strategies have emerged, including chunking, hierarchical aggregation, and efficient architectures like the Longformer and Reformer. These mechanisms now allow models to process documents 10x to 20x longer than the original BERT limit.

The Interpretability Crisis

As neural networks grow in complexity, understanding why a model makes a specific decision becomes critical for ensuring reliability and mitigating bias. The tutorials highlight a shift from "representation analysis" (examining full vectors) to "neuron-level analysis."

By utilizing attribution algorithms such as Integrated Gradients and SHAP, alongside neuron-level probing, researchers can now identify specific "units" within a model responsible for linguistic concepts like gender or syntax. Open-source tools like Captum and NeuroX have matured, making causation analysis a standard part of the modern NLP development workflow.

Structured Knowledge and Graph-based Learning

Despite the dominance of sequence-based transformers, there is a growing recognition that structure still matters. Deep Learning on Graphs (DLG4NLP) involves transforming raw text into structured graphs, such as dependency trees or Abstract Meaning Representations (AMR). Explicitly modeling these relationships through Graph Neural Networks (GNNs) provides a significant performance boost for tasks requiring deep relational understanding, such as semantic parsing.

The Evolution of Evaluation and Data Collection

One of the most significant challenges currently facing the field is the "Evaluation Dilemma." Traditional metrics like BLEU or ROUGE are increasingly viewed as poor proxies for human judgment in generative tasks. The tutorials propose a new taxonomy for organizing metrics into "context-free" vs. "context-dependent" and "trained" vs. "untrained" approaches. The field is rapidly shifting toward trained, context-aware evaluation models that better correlate with human perception.

Furthermore, the volume emphasizes the need for scalable, industrial-grade crowdsourcing. This involves moving toward complex, multi-stage pipelines where human experts and automated systems work in a tight feedback loop. Quality control algorithms, such as the Dawid-Skene model, are essential for ensuring data integrity in large-scale annotation projects.

Key Insights and Future Directions

  • Generalization: There is an urgent need for models that generalize better to new domains with minimal fine-tuning (zero-shot learning).
  • Architecture Debates: For tasks like text ranking, it remains an open question whether it is more effective to modify existing models, adapt them via distillation, or redesign architectures from scratch.
  • Standardization: While many new evaluation metrics have been proposed, the community has yet to settle on a single "perfect" metric for Natural Language Generation that satisfies all use cases.

In summary, the NAACL-HLT 2021 tutorials demonstrate that while transformer-based models have set new benchmarks, the next generation of NLP will be defined by how well we can interpret, structure, and evaluate these powerful tools.

Summary updated

Abstract

The goal of text ranking is to generate an ordered list of texts retrieved from a corpus in response to a query for a particular task. Although the most common formulation of text ranking is search, instances of the task can also be found in many text processing applications. This tutorial provides an overview of text ranking with neural network architectures known as transformers, of which BERT (Bidirectional Encoder Representations from Transformers) (Devlin et al., 2019) is the best-known example. These models produce high quality results across many domains, tasks, and settings.
BibTeX
@inproceedings{yates2021proceedings,
  author = {Andrew Yates and Rodrigo Nogueira and Jimmy Lin and T. B. Brown and Benjamin Mann and Nick Ryder and Melanie Subbiah and Jared Kaplan and Prafulla Dhariwal and Arvind Neelakantan and Pranav Shyam and Girish Sastry and Amanda Askell and Sandhini Agarwal and Ariel Herbert-Voss and Gretchen Krueger and Tom Henighan and Rewon Child and Aditya Ramesh and Daniel M. Ziegler and Jeffrey Wu and Clemens Winter and Christopher Hesse and Mark Chen and Eric J. Sigler and Mateusz Litwin and Scott Gray and Benjamin Chess and J. L. Clark and Christopher Berner and Matthew Henderson and Rami Al‐Rfou and Brian Strope and You Kyung Sung and Hassan Sajjad and Narine Kokhlikyan and Fahim Dalvi and Nadir Durrani and Lingfei Wu and Jennifer cohere.com Jennifer cohere.com and Yu Facebook and Heng Ji and Yunyao Li and Paul School and Daria Baidakova and Valentina Fedorova and Heng Ji and Mitesh M. Khapra and Ananya B. Sai and Dalvi and Fahim and Drutsa and Alexey and Durrani and Valentina Fedorova and Hajishirzi and Ji Heng and Mitesh Khapra and Megorskaya and Olga and Peters and Ananya Sai and Sajjad and Hassan and Ustalov and Dmitry and Wu and Lingfei and Yates and Andrew and Luca Soldaini and Alessandro Moschitti and Ledell Wu and Adam Fisch and Sumit Chopra and Keith Adams and Antoine Bordes and Jason Weston and Zhijing Wu and Jiaxin Mao and Yiqun Liu and Jingtao Zhan and Yukun Zheng and Min Zhang and Shaoping Ma and Hamed Zamani and Mostafa Dehghani and W Croft and Erik Learned-Miller and Jaap Kamps and Zhi Zheng and Kai Hui and Ben He and Xianpei Han and Le Sun and Andrew Yates and Anthony Bau and Yonatan Belinkov and Hassan Sajjad and Nadir Durrani},
  title = {{Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Tutorials}},
  booktitle = {Openalex Unknown},
  year = {2021},
  doi = {10.18653/v1/2021.naacl-tutorials},
}
Figure