Taking Up the Gaokao Challenge: An Information Retrieval Approach

Gong Cheng, Weixi Zhu, Ziwei Wang, Jianghui Chen, Yuzhong Qu

Abstract

The "Gaokao" history exam presents a significant challenge for artificial intelligence due to its multi-sentence stems, classical Chinese quotes, and requirement for external knowledge. We present a three-stage information retrieval (IR) framework—Recollecting Relevant Knowledge, Drawing Evidence, and Reasoning—that utilizes Wikipedia as an external memory. Our system achieves 43.09% accuracy on questions answerable via Wikipedia, significantly outperforming random and traditional IR baselines.

Introduction

The Chinese National College Entrance Exam, or "Gaokao," serves as a benchmark for human intelligence and a "Grand Challenge" for AI. Unlike traditional factoid question-answering (QA) systems that handle single-sentence queries, Gaokao history questions are characterized by multi-sentence stems providing critical context, domain-specific language (including Classical Chinese), and the necessity for external historical facts not present in the prompt. Existing semantic parsing and reading comprehension methods often struggle with these complexities, as they typically assume the answer is contained within a provided text snippet.

Methodology

To address these challenges, we propose a three-stage framework designed to mimic the human process of answering complex historical questions: Recollecting Relevant Knowledge, Drawing Evidence, and Reasoning.

The three-stage IR framework for the Gaokao Challenge.

1. Recollecting Relevant Knowledge

The system first retrieves relevant Wikipedia pages using two primary methods:

  • Concept Pages: Key terms are identified using a "leftmost longest" string matching algorithm. Ambiguity is resolved by cross-referencing other terms in the question stem.
  • Quote Pages: Many questions utilize historical quotes. The system performs exact string searches for these quotes to locate pages providing their specific historical context.

2. Drawing Evidence

To mitigate noise, retrieved pages are ranked and filtered using three strategies:

  • Centrality-based: Pages are ranked by their similarity to a "theme" vector (the average vector of all retrieved pages).
  • Domain-based: Pages not categorized under "history" in Wikipedia’s taxonomy are filtered out.
  • Relevance-based: This measures the relationship between a page, the question stem, and the specific answer options.

3. Reasoning

The final stage employs "relevance-based entailment." The system calculates a support score for each multiple-choice option based on the gathered evidence relative to the stem, selecting the option with the highest score.

Experimental Setup

We evaluated our framework on a dataset of 577 real-life history questions, including 101 from actual Gaokao tests (2005–2014) and 476 from high-quality mock exams. The dataset was split into two subsets:

  • QS-A (123 questions): Confirmed by experts to be answerable using Wikipedia alone.
  • QS-B (454 questions): Requiring specialized resources like textbooks.

Results and Discussion

Our system demonstrated strong retrieval performance, achieving a recall of 0.832 for "gold standard" Wikipedia pages. In the ranking phase, the combination of word-vector centrality, domain filtering, and relevance achieved a peak F-score of 0.406 when retaining the top 6 pages.

QA Accuracy results for QS-A and QS-B datasets compared to a random baseline.

As shown in the performance table, the system reached 43.09% accuracy on the QS-A subset, nearly doubling the 25% random baseline. Even on the full dataset (QS-B), it achieved 31.28%, outperforming traditional BM25 baselines. A key insight was the effectiveness of "Quote Pages"; while Classical Chinese is difficult to parse semantically, the exact strings serve as unique identifiers for specific historical events.

Conclusion

While the IR framework shows promise, several hurdles remain. The system currently struggles with deep reasoning tasks, such as negation and chronological ordering. Furthermore, only about 21% of Gaokao questions are answerable via Wikipedia alone, highlighting the need to integrate specialized domain knowledge, such as history textbooks, into future iterations of the system.

Summary updated

BibTeX
@inproceedings{cheng2016taking,
  author = {Gong Cheng and Weixi Zhu and Ziwei Wang and Jianghui Chen and Yuzhong Qu},
  title = {{Taking Up the Gaokao Challenge: An Information Retrieval Approach}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
}

Loading related content...

Figure