TL;DR
This paper introduces the Tree-guided Stagewise Classifier (TGSC), a novel hierarchical reasoning framework for multi-label, multi-class classification of news articles. By leveraging Large Language Models (LLMs) and a structured, step-by-step approach inspired by "scaffolding theory," TGSC significantly improves classification accuracy in multilingual settings, outperforming baselines without task-specific fine-tuning.
What Problem Does This Paper Address?
The paper tackles the challenge of accurately characterizing and extracting narratives from online news, a crucial task for computational social science and understanding information ecosystems. Specifically, it addresses the complexities of multi-label and multi-class classification in multilingual news analysis, where articles can belong to multiple categories and subcategories. Traditional flat classification approaches often struggle with nuanced hierarchical structures, leading to potential inaccuracies and logical inconsistencies. The SemEval-2025 Task 10, focusing on Entity Framing and Narrative Classification, highlights the need for robust methods that can handle these intricate relationships across different languages.
Methods
The core innovation is the Tree-guided Stagewise Classifier (TGSC). This framework employs a hierarchical reasoning process, mimicking human decision-making by starting with broad categories and progressively narrowing down to specific ones. TGSC leverages the zero-shot capabilities of LLMs by using a Chain-of-Thought (CoT) paradigm.
Key aspects of the method include:
- Hierarchical Reasoning: The classification proceeds through a tree structure, with each node representing a classification level. The model starts at the root and iteratively traverses down the branches, making decisions at each level.
- Scaffolding Theory Inspiration: The approach is inspired by pedagogical scaffolding, where initial broad concepts provide a foundation for learning more specific details.
- Context-Aware Prompt Routing: A dynamic prompt engineering strategy is used. At each level of the hierarchy, validated parent class labels are explicitly integrated into CoT prompts. This "context propagation" guides the LLM's zero-shot inference towards taxonomically valid subclasses.
- Parameter-Efficient Taxonomic Decomposition: The framework strategically decomposes complex taxonomies into LLM-interpretable reasoning chains through structured prompts. This allows for flexible yet precise classification without requiring task-specific adaptations or labeled data.
- No Parametric Gating: Crucially, TGSC achieves hierarchical constraint propagation purely through prompt engineering, without needing to modify LLM parameters or use gating mechanisms. This preserves the LLM's zero-shot flexibility.
Datasets
The experiments were conducted on datasets spanning three languages: English, Portuguese, and Russian. The paper utilized the official test sets provided for SemEval-2025 Task 10.
- Subtask 1 (Entity Framing): 63 English, 57 Russian, and 71 Portuguese samples (191 instances total).
- Subtask 2 (Narrative Classification): 101 English, 60 Russian, and 100 Portuguese samples (261 instances total).
The official evaluation metrics used were Exact Match Ratio (EMR) and Sample F1.
Key Insights and Ideas
- Hierarchical Reasoning is Key: The most significant insight is that a structured, hierarchical approach dramatically improves classification performance, especially for complex, multi-label tasks. Decomposing the problem into stages allows LLMs to focus on specific aspects at each level, leading to more accurate and consistent results.
- Prompt Engineering for Structure: The paper demonstrates that sophisticated prompt engineering can effectively impose hierarchical constraints on LLMs without altering their underlying architecture or requiring extensive fine-tuning. This is a powerful way to leverage LLMs for structured reasoning.
- LLMs Excel at Global Patterns: The results suggest that LLMs are particularly adept at capturing global semantic patterns in news articles. However, they may struggle with fine-grained, localized reasoning, indicating a potential area for future improvement.
- Scaffolding Theory in AI: Applying principles from educational psychology, like scaffolding, can provide effective frameworks for complex AI tasks, guiding models through progressive learning stages.
Key Results
The TGSC framework achieved competitive performance across all three languages for both Subtask 1 and Subtask 2.
- Subtask 1 Rankings: 15th (English), 13th (Portuguese), and 12th (Russian).
- Subtask 2 Rankings: 8th (English), 8th (Portuguese), and 5th (Russian).
- Ablation Study: The hierarchical stagewise approach consistently outperformed a simplified single-stage variant, highlighting the effectiveness of the proposed method. For example, in Subtask 2, TGSC achieved Sample F1 scores of 0.321 (en), 0.266 (pt), and 0.335 (ru), while the variant without stagewise classification scored 0.196 (en), 0.227 (pt), and 0.148 (ru).
- Open-Source LLMs: Experiments with open-source LLMs also showed substantial performance improvements with stagewise classification, with baselines often performing at or near zero without it.
Limitations and Future Work
The authors acknowledge a potential limitation: the hierarchical structure introduces a risk of error propagation. Misclassifications at parent nodes could negatively influence subsequent child-level predictions.
Future work suggested includes:
- Confidence-Based Re-evaluation: Developing mechanisms to re-evaluate predictions based on confidence scores.
- Multi-Path Traversal: Exploring alternative paths through the hierarchy to mitigate error propagation.
- Uncertainty-Aware Prompting: Using prompting strategies that explicitly handle uncertainty in predictions.
- Improving Local Reasoning: Enhancing LLMs' ability to accurately extract and reason over local information, addressing the observed weakness in fine-grained analysis.
Summary updated