Abstract
The "Gaokao" history exam presents a significant challenge for artificial intelligence due to its multi-sentence stems, classical Chinese quotes, and requirement for external knowledge. We present a three-stage information retrieval (IR) framework—Recollecting Relevant Knowledge, Drawing Evidence, and Reasoning—that utilizes Wikipedia as an external memory. Our system achieves 43.09% accuracy on questions answerable via Wikipedia, significantly outperforming random and traditional IR baselines.
Introduction
The Chinese National College Entrance Exam, or "Gaokao," serves as a benchmark for human intelligence and a "Grand Challenge" for AI. Unlike traditional factoid question-answering (QA) systems that handle single-sentence queries, Gaokao history questions are characterized by multi-sentence stems providing critical context, domain-specific language (including Classical Chinese), and the necessity for external historical facts not present in the prompt. Existing semantic parsing and reading comprehension methods often struggle with these complexities, as they typically assume the answer is contained within a provided text snippet.
Methodology
To address these challenges, we propose a three-stage framework designed to mimic the human process of answering complex historical questions: Recollecting Relevant Knowledge, Drawing Evidence, and Reasoning.
1. Recollecting Relevant Knowledge
The system first retrieves relevant Wikipedia pages using two primary methods:
- Concept Pages: Key terms are identified using a "leftmost longest" string matching algorithm. Ambiguity is resolved by cross-referencing other terms in the question stem.
- Quote Pages: Many questions utilize historical quotes. The system performs exact string searches for these quotes to locate pages providing their specific historical context.
2. Drawing Evidence
To mitigate noise, retrieved pages are ranked and filtered using three strategies:
- Centrality-based: Pages are ranked by their similarity to a "theme" vector (the average vector of all retrieved pages).
- Domain-based: Pages not categorized under "history" in Wikipedia’s taxonomy are filtered out.
- Relevance-based: This measures the relationship between a page, the question stem, and the specific answer options.
3. Reasoning
The final stage employs "relevance-based entailment." The system calculates a support score for each multiple-choice option based on the gathered evidence relative to the stem, selecting the option with the highest score.
Experimental Setup
We evaluated our framework on a dataset of 577 real-life history questions, including 101 from actual Gaokao tests (2005–2014) and 476 from high-quality mock exams. The dataset was split into two subsets:
- QS-A (123 questions): Confirmed by experts to be answerable using Wikipedia alone.
- QS-B (454 questions): Requiring specialized resources like textbooks.
Results and Discussion
Our system demonstrated strong retrieval performance, achieving a recall of 0.832 for "gold standard" Wikipedia pages. In the ranking phase, the combination of word-vector centrality, domain filtering, and relevance achieved a peak F-score of 0.406 when retaining the top 6 pages.
As shown in the performance table, the system reached 43.09% accuracy on the QS-A subset, nearly doubling the 25% random baseline. Even on the full dataset (QS-B), it achieved 31.28%, outperforming traditional BM25 baselines. A key insight was the effectiveness of "Quote Pages"; while Classical Chinese is difficult to parse semantically, the exact strings serve as unique identifiers for specific historical events.
Conclusion
While the IR framework shows promise, several hurdles remain. The system currently struggles with deep reasoning tasks, such as negation and chronological ordering. Furthermore, only about 21% of Gaokao questions are answerable via Wikipedia alone, highlighting the need to integrate specialized domain knowledge, such as history textbooks, into future iterations of the system.
Summary updated