CLICKER: A Computational LInguistics Classification Scheme for Educational Resources

Swapnil Hingmire, Irene Li, Rena Kawamura, Benjamin Chen, Alexander R. Fabbri, Xiangru Tang, Yixin Liu, Thomas George, Tammy Liao, Wai Pan Wong, Vanessa Yan, Richard Zhou, Girish K. Palshikar, Dragomir R. RadevarXiv 2021 · OpenReview

TL;DR

Researchers have developed CLICKER, a new classification scheme specifically for Computational Linguistics (CL) and Natural Language Processing (NLP) educational resources. This taxonomy, derived from university course materials, aims to fill a critical gap in organizing and accessing CL/NLP knowledge, enabling better resource discovery, recommendation, and learning path generation.

What Problem Does This Paper Address?

The field of Computational Linguistics (CL) and Natural Language Processing (NLP) is rapidly expanding, yet it lacks a comprehensive and standardized classification system akin to those found in other scientific domains like mathematics (MSC) or general computer science (ACM-CCS). This absence makes it challenging to effectively organize, search, and access the vast and growing body of educational materials and research papers in CL/NLP. Existing schemes are either too broad, too narrow, or not specifically tailored to the nuances of CL/NLP topics, leading to difficulties in tasks like finding relevant course materials, understanding prerequisite relationships, or building intelligent tutoring systems.

Methods

The authors propose CLICKER, a classification scheme for CL/NLP, built through a multi-stage process:

  1. Candidate Keyword Extraction: They began by analyzing online lecture notes from 77 university courses. Keywords and phrases were extracted from lecture slide titles and headers using techniques like TextRank. They also processed resources from the TutorialBank corpus.
  2. Topic List Combination and Filtering: Candidate topics from different sources were combined, and those appearing at least three times were retained, resulting in an initial list of 465 candidates.
  3. Manual Filtering and Grouping: The list underwent manual filtering to remove spurious entries and group similar concepts. The final taxonomy, CLICKER v1.0, comprises 271 topics, covering core CL/NLP areas, related AI topics, and essential mathematical/statistical concepts. The taxonomy is presented as a hierarchical structure, with an initial focus on educational aspects.

Datasets

  • Online Lecture Notes: Data from 77 university courses on CL/NLP were analyzed.
  • LectureBank (Li et al., 2019): Used for initial keyword extraction and as a source for training concept embeddings.
  • TutorialBank (Fabbri et al., 2018): Also used for keyword extraction from resource titles.
  • ACL Anthology (AA): Mentioned as a primary source of research papers that would benefit from such a classification scheme.
  • AAN (aan.how): A platform that has manually assigned 23,293 resources to nodes of the proposed taxonomy for demonstration purposes.

Key Insights and Ideas

  • The Need for a CL/NLP Specific Taxonomy: Existing broad classification systems (like ACM-CCS) have limited depth and coverage for the specific subfields within CL/NLP, missing key topics like "Summarization" or "Question Answering."
  • Educational Resources as a Rich Source: University lecture notes and tutorials provide a clean and fine-grained source of topics relevant to learning the field.
  • CLICKER's Potential for Diverse Applications: The taxonomy is not just for indexing but can power advanced educational tools like intelligent tutoring platforms, personalized resource recommendation, prerequisite chain learning, and automated survey generation.
  • Prerequisite Learning is Key: Understanding the dependencies between concepts is crucial for effective learning, and a structured taxonomy like CLICKER can facilitate this.

Key Results

  • The CLICKER taxonomy (v1.0) contains 271 topics, focusing on educational aspects of CL/NLP.
  • The paper demonstrates the utility of CLICKER in five applications:
  • Educational Platform (AAN): Resources are organized and browsable via the taxonomy's hierarchical structure.
  • Resource Retrieval: The taxonomy can enhance search results by providing topic context.
  • Resource Recommendation: Given a project description, CLICKER can suggest relevant concepts and resources.
  • Prerequisite Chain Learning: The taxonomy's structure aids in learning concept dependencies, with experiments showing promising results for predicting prerequisite relationships between NLP and Computer Vision concepts using a neural network model.
  • Survey Generation: CLICKER can be used to generate concise surveys on specific topics by leveraging existing resources.

Limitations and Future Work

  • Taxonomy Depth and Coverage: While CLICKER aims for comprehensive coverage, the authors acknowledge that it is an ongoing effort and may not capture every niche topic. The current version has 271 topics, which is significantly smaller than some existing broad taxonomies but more focused on CL/NLP.
  • Manual Filtering: The process involved manual filtering, which can be subjective and time-consuming.
  • Accuracy of Applications: The paper notes that the resource recommendation system, while promising, can sometimes miss important keywords. Future work includes improving keyword extraction algorithms.
  • Survey Generation Accuracy: Preliminary results for survey generation show some inaccuracies and irrelevant content, indicating a need for further refinement of the summarization models.
  • Future Directions: The authors suggest improving the recommendation system, enhancing survey generation quality, and further expanding the taxonomy's coverage and depth. They also highlight the potential for using CLICKER to analyze the evolution of research trends within CL/NLP.

Summary updated

Abstract

A classification scheme of a scientific subject gives an overview of its body of knowledge. It can also be used to facilitate access to research articles and other materials related to the subject. For example, the ACM Computing Classification System (CCS) is used in the ACM Digital Library search interface and also for indexing computer science papers. We observed that a comprehensive classification system like CCS or Mathematics Subject Classification (MSC) does not exist for Computational Linguistics (CL) and Natural Language Processing (NLP). We propose a classification scheme -- CLICKER for CL/NLP based on the analysis of online lectures from 77 university courses on this subject. The currently proposed taxonomy includes 334 topics and focuses on educational aspects of CL/NLP; it is based primarily, but not exclusively, on lecture notes from NLP courses. We discuss how such a taxonomy can help in various real-world applications, including tutoring platforms, resource retrieval, resource recommendation, prerequisite chain learning, and survey generation.
BibTeX
@misc{hingmire2021clicker,
  author = {Swapnil Hingmire and Irene Li and Rena Kawamura and Benjamin Chen and Alexander R. Fabbri and Xiangru Tang and Yixin Liu and Thomas George and Tammy Liao and Wai Pan Wong and Vanessa Yan and Richard Zhou and Girish K. Palshikar and Dragomir R. Radev},
  title = {{CLICKER: A Computational LInguistics Classification Scheme for Educational Resources}},
  year = {2021},
  eprint = {2112.08578},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.08578},
}
Figure