TL;DR
This paper tackles the challenge of navigating the rapidly growing landscape of Large Language Model (LLM) research by proposing an automated method to classify LLM survey papers into a structured taxonomy. Their approach, leveraging graph representation learning on co-category graphs, significantly outperforms traditional language model approaches and even human performance, offering a promising solution for researchers to efficiently discover relevant survey literature.
What Problem Does This Paper Address?
The explosion of research in Large Language Models (LLMs) has led to a deluge of survey papers aiming to synthesize this ever-expanding body of knowledge. While invaluable for newcomers and established researchers alike, the sheer volume of these surveys makes it increasingly difficult to keep pace and identify the most relevant ones. Existing methods for organizing research, like arXiv categories, often lack the granularity needed to distinguish between papers focusing on different aspects of LLMs, and the textual similarity between survey titles and abstracts further complicates automated classification. This paper addresses the need for an automated system that can effectively categorize LLM survey papers, making it easier for researchers to navigate and understand the field.
Methods
The core of this paper's contribution lies in its novel application of Graph Representation Learning (GRL) for taxonomy classification. Instead of relying solely on text, they construct attributed graphs where nodes represent papers and edges capture relationships. They explore three types of graphs:
- Text Graph: Connects papers to words based on their occurrence in titles and abstracts, and also connects words that co-occur within a text corpus.
- Co-author Graph: Links papers if they share at least one author.
- Co-category Graph: Connects papers that share at least one arXiv category.
These graphs are then fed into Graph Neural Networks (GNNs), specifically a two-layer GCN, to learn representations that capture both the textual content (via TF-IDF features) and the structural relationships within the graph. They compare this GRL approach against two main paradigms:
- Fine-tuning pre-trained language models (LLMs) on the survey paper text.
- Zero-shot/Few-shot classification using LLMs.
A key aspect of their methodology is the exploration of weak-to-strong generalization, where they investigate fine-tuning LLMs using "weak labels" generated by their GNN model, rather than relying solely on human-annotated "ground-truth" labels.
Datasets
The authors collected and analyzed 144 LLM survey papers. They scraped metadata from arXiv and supplemented it with information from Google Scholar and the ACL Anthology. The papers span from July 2021 to January 2024.
They introduced a new taxonomy specifically designed for LLM survey papers, which is visualized in Figure 2 of the paper. This taxonomy is hierarchical, with broad categories like "Applications" and "Model Techniques," further subdividing into more specific leaf nodes.
For evaluation, they created three subsets of their data:
- Data Nov23: Survey papers collected up to November 2023.
- Data Jan24: Survey papers collected up to January 2024, including a new category "Hardware Architecture."
- Datasubset: A subset of Data Jan24 where less frequent categories (Law, Finance, Education, Hardware Architecture, Others) were removed to create a more balanced dataset.
Key Insights and Ideas
- Graph Structure is Key: The most significant insight is that leveraging the structural information of co-category graphs, which capture relationships based on shared arXiv categories, dramatically improves taxonomy classification performance. This suggests that how papers are related through existing metadata is more informative than just their text content for this task.
- Beyond Textual Similarity: The paper highlights that LLM survey papers often use similar terminology, making text-based classification challenging. GRL, by incorporating graph structure, can overcome this limitation.
- Weak Labels Can Be Powerful: The finding that fine-tuning LLMs with weak labels generated by a GNN can outperform training with ground-truth labels is a crucial insight. This demonstrates the potential of "weak-to-strong generalization," where a simpler model can generate valuable training signals for more complex models.
- GRL Outperforms LLMs: The GRL approach consistently outperformed both fine-tuned LLMs and zero-shot/few-shot LLM classifications, even surpassing average human performance on this task.
Key Results
- Co-category Graphs Shine: In the Data Jan24 subset, the co-category graph approach achieved an accuracy of 75.17% and a weighted F1 score of 74.60%, significantly outperforming text graphs (20.00% accuracy, 19.24% F1) and co-author graphs (29.63% accuracy, 29.24% F1).
- Surpassing Human Performance: The GRL method, particularly with co-category graphs, achieved an accuracy of 79.26% and a weighted F1 score of 77.88% on the Datasubset, which is noted to be better than the average human recognition level.
- Weak Labels Boost LLMs: Fine-tuning LLMs with weak labels generated by a GCN resulted in higher accuracy and weighted F1 scores compared to fine-tuning with ground-truth labels. For example, BERT fine-tuned with weak labels achieved 58.74% accuracy, while with ground-truth labels it was 30.43% on the Data Nov23 subset.
- LLMs Struggle with Taxonomy: Even with few-shot prompting, LLMs like GPT-3.5 and GPT-4 struggled to match human performance in classifying papers into the proposed taxonomy, highlighting the complexity of the task and the limitations of current LLM capabilities in this specific domain.
Limitations and Future Work
- Graph Sparsity: The authors acknowledge that constructing graph structures can be challenging, especially when papers come from very distinct fields, leading to sparse graphs that may weaken GRL effectiveness.
- Taxonomy Design: Future work could explore different taxonomy designs, such as adopting existing ontologies or allowing multiple class assignments per paper.
- GPT-based Applications: A key future direction is to tailor GPT-based applications to assist readers in understanding survey papers more effectively, building upon the insights gained from this work.
- Domain Shift for LLMs: The paper notes that smaller LLMs can be sensitive to domain shifts, and larger models might overfit on smaller datasets when fine-tuned, suggesting a need for more robust fine-tuning strategies.
Summary updated