Introduction of the HTRDP Chinese IR Evaluation Task

Le Sun, Junlin ZhangEVIA@NTCIR 2007 · OpenReview

Abstract

The HTRDP Chinese Information Retrieval (IR) Evaluation is a benchmark initiative designed to advance large-scale Chinese search technology. Much like the TREC or NTCIR workshops, it provides standardized datasets, query topics, and evaluation metrics necessary to measure how effectively different algorithms navigate the unique linguistic complexities of the Chinese web. This paper outlines the evolution of the task, the methodology employed, and the performance of participating systems on web-scale data.

Introduction

In the early 2000s, while English-language search engines were maturing rapidly, Chinese information retrieval faced distinct hurdles. The language lacks explicit word boundaries, requiring complex segmentation, features a high density of named entities, and uses a character-based system that makes standard keyword matching less effective. To address these challenges, the HTRDP (sponsored by China's "863 Program") established a series of annual evaluations to move Chinese IR from small-scale academic exercises to robust, web-scale performance.

Methodology

The evaluation framework follows the classic "Cranfield" paradigm used by international benchmarks like TREC. The tasks evolved from simple document retrieval on small corpora (200MB) in 2003 to "Relevant Web Page Retrieval" on a 100GB web crawl by 2005.

Because the 2005 dataset was too large for humans to judge every single page for every query, the organizers used a pooling technique. They took the top 100 results from all participating systems, merged them, and had human experts judge only that subset for relevance. Query topics were structured in XML with four parts: an identifier, a title (short keywords), a description (1-2 sentences), and a narrative (detailed intent). Systems were ranked using standard IR metrics: Mean Average Precision (MAP), R-Precision, and Precision at 10 documents (P@10).

Dataset Evolution

The evaluation's data grew significantly over three years:

  • 2003: Small (200MB) and Large (2GB) corpora.
  • 2004: A 15GB web corpus.
  • 2005 (CWT100g): The "Chinese Web Test" collection, provided by Peking University. This was a significant leap, containing 5,712,710 web pages (~100GB) crawled from over 17,000 Chinese websites. This dataset provided not just text, but also link information and anchor text, allowing for more modern link-analysis algorithms.

Results and Analysis

Performance improved significantly between 2003 and 2005 as groups used previous years' evaluation data as training sets to tune system parameters and refine named entity recognition. However, linguistic bottlenecks like segmentation errors remain a primary point of failure. If a system incorrectly splits a person's name or a new technical abbreviation into separate characters, retrieval accuracy plummets.

A key finding was the performance gap between manual and automatic queries. In 2005, systems using manually formed queries (where a human interprets the topic) consistently outperformed those using automatically generated queries.

Manual Query Performance

The top-performing system in the manual query task (System 5) achieved a MAP of 0.3671 and a P@10 of 0.7040, meaning 7 out of the first 10 results were relevant on average.

Performance metrics for manual queries in the 2005 evaluation.

Automatic Query Performance

The best automatic system (System 4) reached a MAP of 0.3175. While these scores were higher than previous years, the jump was largely due to the inclusion of link analysis and anchor text, which became possible once the corpus reached a web-scale size of 100GB.

Performance metrics for automatic queries in the 2005 evaluation.

Conclusion and Future Work

Despite progress, systems still struggle with semantic mismatch (synonymy) and finding optimal weighting schemes for query words. Future work planned by the organizers included doubling the corpus size to 200GB and moving away from manually designed topics toward using real user query logs to better simulate actual search behavior.

Summary updated

BibTeX
@inproceedings{sun2007introduction,
  author = {Le Sun and Junlin Zhang},
  title = {{Introduction of the HTRDP Chinese IR Evaluation Task}},
  booktitle = {EVIA@NTCIR},
  year = {2007},
  url = {https://openreview.net/forum?id=9MhsnhK517},
}
Figure