TL;DR
This paper introduces Tailored Progressive Curriculum Learning (TPCL), a novel training strategy for Attributed Question Answering (AQA) models. TPCL addresses the challenge of simultaneously training AQA models on evidence identification, cross-source relation recognition, and anti-distraction reasoning by progressively introducing complexity and tailoring training curricula. This approach leads to significant performance improvements, outperforming existing methods on the QuoteSum benchmark.
What Problem Does This Paper Address?
Attributed Question Answering (AQA) is a task where models must generate long-form answers that include evidence extracted from provided source documents, along with free-style contextual information. A key challenge in AQA is that models need to simultaneously master several distinct capabilities: identifying relevant evidence, understanding relationships between evidence from different sources, and resisting distraction from irrelevant information. Current training methods often apply a uniform learning process, which is insufficient for developing these specialized skills effectively. This paper aims to bridge this gap by proposing a more sophisticated training methodology that caters to the unique learning dynamics of each AQA capability.
Methods
The core contribution of this paper is the Tailored Progressive Curriculum Learning (TPCL) approach. TPCL is built upon three main components:
- Tailored Programme Construction: The authors define three distinct training programs (PM1, PM2, PM3) to target specific AQA capabilities:
- PM1: Focuses on evidence identification by training the model to predict evidence from reliable sources given a question.
- PM2: Enhances cross-source relation recognition by training the model to analyze relationships between evidence and generate the full answer.
- PM3: Improves anti-distraction reasoning by training the model to generate accurate answers even when presented with distracting, irrelevant sources.
- Complexity-Based Curriculum Development: TPCL constructs curricula that are not only tailored but also controllable in complexity. This involves:
- Creating a hybrid training set by combining instances from PM1, PM2, and PM3.
- Computing a complexity score for each instance based on factors like answer length, number of sources, and entity density.
- Ranking instances by complexity and partitioning them into complexity-controllable curricula for training.
- Curriculum Learning: TPCL employs two curriculum learning strategies:
- Progressive Curriculum Learning: For encoder-decoder models, this strategy uses the EXP3 algorithm to dynamically select curricula, gradually increasing complexity. It incorporates a dynamic smoothing factor and a "jogging-like" curriculum updating mechanism to ensure stable transitions and prevent catastrophic forgetting.
- One-off Curriculum Learning: For decoder-only LLMs, a simplified version is used where a single curriculum is formed by randomly selecting instances from the complexity-aware list, and this curriculum is used for in-context learning.
Datasets
The primary dataset used for training and evaluation is QuoteSum (Schuster et al., 2024). The authors also construct a challenge test set to specifically evaluate the anti-distraction capabilities of AQA models. This challenge set features a larger number of paragraph-level sources and uses GPT-40 to generate noisy, yet highly relevant, distracting sources.
Key Insights and Ideas
- Decomposition is Key: Breaking down the complex AQA task into specialized sub-skills (evidence identification, relation recognition, anti-distraction) and training them through tailored programs is crucial for overall model improvement.
- Complexity Matters: A carefully constructed curriculum that progressively increases difficulty, based on quantifiable metrics like answer length and source count, leads to more stable and effective learning.
- Dynamic Curriculum Selection: Using adaptive algorithms like EXP3 to select curricula based on the model's current ability is more effective than a static curriculum.
- Anti-Distraction is a Differentiator: The proposed TPCL approach shows a marked improvement in handling distracting sources, a critical aspect for real-world AQA systems.
- Decoder-Only LLMs Benefit Differently: While encoder-decoder models show consistent degradation on the challenge set, decoder-only LLMs exhibit some improvements, suggesting their ability to leverage topic-relevant sources even without explicit answers.
Key Results
- TPCL achieves substantial improvements on the QuoteSum benchmark, reaching a 73.9% Sem-F1 score with the T5-large backbone.
- The full version of TPCL consistently outperforms baseline SEMQA models across various backbones (BART-large, T5-large, LLaMA-3.1, Gemma-3.0).
- TPCL outperforms another curriculum learning method, ICL-SC, on all metrics.
- The combination of T5-large and TPCL yields a higher F1-score than the strong GPT-4 based CoTAR.
- Ablation studies confirm that both dynamic smoothing factors and jogging-like curriculum updating positively impact performance.
- On the challenging test set, TPCL exhibits better anti-distraction capabilities, showing less performance degradation compared to SEMQA.
Limitations and Future Work
- Single-Round Demonstration Ordering: For in-context learning with decoder-only LLMs, the current approach uses a single round of demonstration ordering. The authors suggest exploring an iterative curriculum refinement framework where the model autonomously diagnoses curricula through self-improving mechanisms.
- GPT-4 Oriented ICL: The paper mentions exploring a GPT-4 oriented ICL method using TPCL as a future direction.
- Larger Models: The current work doesn't specifically explore TPCL's potential on larger-sized models like GPT-4, which is an area for future investigation.
Summary updated