SEUNIPD@CLEF: Team JIHUMING on Enhancing Search Engine Performance with Character N-Grams, Query Expansion, and Named Entity Recognition

Isil Atabek, Huimin Chen, Jesús Moncada-Ramírez, Nicolò Santini, Giovanni Zago, Nicola FerroCLEF (Working Notes) 2023

Abstract

This paper details the participation of Team JIHUMING in the CLEF 2023 LongEval Lab, which focuses on the temporal persistence of Information Retrieval (IR) systems. As web content and user language evolve, search engines must maintain effectiveness over time. Our study evaluates a variety of Natural Language Processing (NLP) enhancements—including character N-grams, WordNet-based query expansion, and Named Entity Recognition (NER)—within a Lucene-based BM25 framework. Using the LongEval dataset, we demonstrate that while larger character N-grams significantly improve retrieval performance, complex features like NER can introduce noise and increase computational overhead without providing a corresponding boost in accuracy. Our findings highlight the importance of native language processing and the potential pitfalls of over-engineering IR pipelines.

Introduction

Most Information Retrieval (IR) systems are evaluated using static snapshots of data, failing to account for the dynamic nature of the web. Language evolves, new topics emerge, and the relevance of documents shifts over weeks or months. This study addresses the challenge of temporal persistence: measuring how well a search system built on current data performs on queries and documents from the future.

Participating in the CLEF 2023 LongEval Lab, we investigated how various NLP enhancements affect a system's ability to remain effective across short-term (one month) and long-term (two months) horizons. The evaluation was conducted in a bilingual (English and French) environment, providing a robust testbed for cross-lingual and native-language retrieval strategies.

Methodology

Our search engine was developed using Java and the Lucene library, utilizing the BM25 similarity function as the primary ranking mechanism. The experimental pipeline included several standard and experimental components:

  • Pre-processing: We employed whitespace tokenization, lowercasing, and stopword removal. For English, we used the Terrier stopword list, while a custom list was developed for French.
  • Stemming: Minimal stemmers (Harman for English, Savoy for French) were used to reduce words to their roots while avoiding the loss of meaning associated with over-truncation.
  • Character N-Grams: To capture recurring sub-word structures such as prefixes and suffixes, we generated N-grams of sizes 3, 4, and 5.
  • Query Expansion: For English queries, we integrated WordNet to automatically append synonyms, aiming to broaden the search scope and mitigate vocabulary mismatch.
  • Named Entity Recognition (NER): Using Apache OpenNLP, we extracted locations, person names, and organizations to populate a specialized index field, testing whether entity-aware indexing improves precision.
  • Multilingual Indexing: We implemented a MultilingualDirectoryIndexer to combine English and French versions of documents into single Lucene records, allowing for unified retrieval.

Experimental Setup

The study utilized the LongEval collection, derived from real-world data from the French search engine Qwant.

  • Training Set: Approximately 1.57 million documents and 672 queries.
  • Test Sets: Two distinct sets gathered in July 2022 (short-term) and September 2022 (long-term).
  • Bilingual Nature: The collection includes original French documents and queries alongside English translations generated via the CUBBITT system.

Results and Discussion

Our evaluation focused on Mean Average Precision (MAP), Normalized Discounted Cumulative Gain (NDCG), and R-precision (Rprec). Several key insights emerged from the experimental runs:

The "Native" Advantage

Despite the availability of high-quality English translations, runs using French queries on French documents consistently outperformed their English counterparts. This suggests that the nuances lost during machine translation significantly impact retrieval precision, favoring native-language pipelines.

The Impact of N-Gram Size

Effectiveness generally increased with N-gram size. While 3-grams capture local sub-word information, 5-grams provided sufficient context to significantly improve MAP. The 5-gram configuration allowed the system to better handle morphological variations and spelling inconsistencies.

The NER Paradox and Computational Cost

One of the most significant findings was that NER negatively affected scores. While often considered a sophisticated NLP addition, NER likely introduced noise or over-constrained the search space in this specific IR task, leading to the lowest scores in the study (e.g., Run 06_en_en_4gram_ner achieved a MAP of only 0.0360).

Furthermore, the computational cost was prohibitive. While a standard index took approximately 1 hour to build, the NER-inclusive index required 16 hours of processing time. This represents a massive increase in resource consumption for a decrease in search quality.

Statistical Significance

Using Two-Way ANOVA and Tukey’s HSD test, we confirmed that the performance gap between N-gram-heavy systems and NER-enhanced systems was statistically significant, validating that simpler sub-word features were superior to complex entity extraction in this context.

Conclusion

Our participation in LongEval 2023 demonstrates that "more features" do not always equate to better performance. While character N-grams (specifically 5-grams) are highly effective for maintaining temporal persistence, the integration of NER can be counterproductive both in terms of accuracy and efficiency.

Future work will explore:

  1. Increasing N: Investigating the limits of larger N-gram sizes.
  2. Refined NLP Integration: Developing methods to use NER as a re-ranking feature rather than a primary indexing field.
  3. Neural IR: Transitioning toward machine-learning-based ranking models that can dynamically adapt to temporal shifts in document and query distributions.

Summary updated

BibTeX
@inproceedings{atabek2023seunipd,
  author = {Isil Atabek and Huimin Chen and Jesús Moncada-Ramírez and Nicolò Santini and Giovanni Zago and Nicola Ferro},
  title = {{SEUNIPD@CLEF: Team JIHUMING on Enhancing Search Engine Performance with Character N-Grams, Query Expansion, and Named Entity Recognition}},
  booktitle = {CLEF (Working Notes)},
  year = {2023},
}
Figure