OVERVIEW OF THE TREC 2023 PRODUCT PRODUCT SEARCH TRACK

Daniel Campos, Surya Kallumadi, Corby Rosset, Cheng Xiang Zhai, Alessandro Magnani

Abstract

The inaugural TREC 2023 Product Search Track addresses the unique challenges of e-commerce retrieval, where traditional web search paradigms often fall short. By introducing a large-scale, multi-modal dataset of over 1.6 million products, the track provides a benchmark for evaluating end-to-end retrieval systems. Results indicate that traditional sparse retrieval methods like BM25 remain highly competitive against modern neural dense retrievers, particularly in zero-shot settings. This paper summarizes the track's methodology, the characteristics of the new corpus, and key insights into the performance of various retrieval architectures.

1. Introduction

Product search represents a specialized domain within information retrieval. Unlike general web search, which primarily deals with unstructured text, product search involves finding specific items characterized by a mix of text descriptions, structured metadata (e.g., dimensions, color, brand), and visual information. Users often search with high-precision intent, such as "waterproof running shoe," requiring systems to navigate complex attribute relationships.

Historically, datasets like the Amazon ESCI (Shopping Queries) dataset have facilitated research in product re-ranking. However, there has been a notable lack of benchmarks for the initial retrieval stage—identifying relevant items from a massive catalog. The TREC 2023 Product Search Track was established to bridge this gap, offering a multi-modal corpus and a standardized evaluation framework to compare traditional and neural retrieval methods.

2. The Product Search Track Corpus

The track introduced a significant expansion of the ESCI dataset, designed to support large-scale retrieval tasks.

  • Scale: The corpus contains 1,661,907 unique products.
  • Multi-modality: Over 90% of the products include image data (provided as 64x64 thumbnails), enabling research into multi-modal retrieval.
  • Metadata Enrichment: Products are enriched with contextual metadata, including customer reviews, technical specifications, and hierarchical category information crawled from Amazon.
  • Query Set: The evaluation utilized a test set of 926 queries. Of these, 182 queries received deep manual relevance judgments from NIST (National Institute of Standards and Technology) assessors on a four-point scale.

3. Methodology

To establish a robust benchmark, the track organizers implemented several query generation and evaluation strategies:

  • Query Generation: Beyond existing user queries, GPT-4 was used to generate synthetic queries based on product descriptions. Additionally, "high-overlap" queries were created by extracting specific sub-spans from product titles to simulate keyword-heavy user behavior.
  • Evaluation Framework: NIST assessors provided ground-truth labels, ranging from "Irrelevant" to "Perfectly Relevant."
  • Baseline Systems: The organizers provided 39 baseline runs, encompassing traditional BM25, single-stage dense retrievers (e.g., BGE, GTE, E5), and hybrid systems combining sparse and dense approaches.

4. Key Insights and Results

The 2023 track yielded several surprising findings regarding the efficacy of modern retrieval models in the e-commerce domain.

4.1 The Persistence of BM25

Traditional BM25 retrieval demonstrated remarkable resilience. In many scenarios, it outperformed general-purpose pretrained neural embedding models. This suggests that the keyword-centric nature of product search—where specific model numbers or technical terms are critical—remains a significant challenge for dense models that map queries into abstract vector spaces.

4.2 Challenges in Zero-Shot Dense Retrieval

Many popular dense retrieval models (such as BGE and GTE) struggled in zero-shot settings. Interestingly, the common trend in general text benchmarks (like MTEB), where larger models consistently outperform smaller ones, did not hold true here. Some large models performed poorly, scoring near zero on complex, specific queries.

4.3 The Role of Metadata

The inclusion of metadata (reviews and specifications) presented a trade-off. While it generally improved Recall@100 by providing more "surface area" for matching, it frequently degraded NDCG@10. The additional text often introduced noise, distracting models from the core product features and making it harder to rank the most relevant items at the very top.

4.4 Fine-Tuning Nuances

Fine-tuning results were mixed. Naive fine-tuning recipes actually caused larger models to perform worse. Conversely, smaller models benefited significantly from fine-tuning, evolving from poor performers to highly competitive systems.

4.5 Top Performers

The most effective system was a hybrid model (f_splade_bm25), which integrated SPLADE (a neural sparse retriever) with BM25, achieving an NDCG@10 of 0.7505. For comparison, the standard BM25 baseline achieved a score of 0.6540.

5. Conclusion and Future Work

The TREC 2023 Product Search Track highlights that while neural methods are advancing, they have not yet rendered traditional sparse retrieval obsolete in the e-commerce sector. The "keyword-heavy" nature of product queries continues to favor methods that prioritize exact term matching.

Future work will focus on:

  1. Effective Multi-modal Integration: Leveraging the provided image data more effectively than current text-centric models.
  2. Domain-Specific Fine-Tuning: Developing specialized training recipes for product-specific data.
  3. Generative Retrieval: Exploring the role of Large Language Models (LLMs) in the retrieval phase rather than just re-ranking.

For practitioners, the primary takeaway is that BM25 remains a formidable baseline in product search, and hybrid approaches currently offer the best balance of keyword precision and semantic depth.

Summary updated

BibTeX
@inproceedings{campos2023overview,
  author = {Daniel Campos and Surya Kallumadi and Corby Rosset and Cheng Xiang Zhai and Alessandro Magnani},
  title = {{OVERVIEW OF THE TREC 2023 PRODUCT PRODUCT SEARCH TRACK}},
  booktitle = {TREC},
  year = {2023},
}
Figure