TL;DR
This paper introduces TripClick, a large-scale dataset of health web search engine click logs and a corresponding benchmark. This resource is crucial for training and evaluating advanced neural information retrieval (IR) models in the health domain, an area previously lacking such comprehensive data. Experiments show that neural models significantly outperform traditional methods on this benchmark, especially for common queries.
What Problem Does This Paper Address?
Developing effective information retrieval (IR) systems, particularly those using complex neural architectures, requires vast amounts of training data. While click logs from general web search engines have been extensively used for this purpose, the health domain has been underserved. Health-related information needs are often complex and nuanced, involving diagnostic, therapeutic, or educational queries. Furthermore, users of health search engines are frequently healthcare professionals, whose search behaviors might differ from the general public. Existing health IR benchmarks are typically small-scale, lacking the richness and volume needed to train modern deep learning models. This paper addresses this gap by releasing a large-scale click log dataset from a specialized health web search engine and creating a benchmark from it.
Methods
The core contribution is the creation of the TripClick dataset and benchmark.
The TripClick dataset comprises approximately 5.2 million user interactions from the Trip Database health web search engine, collected between 2013 and 2020. Each log entry includes session ID, query, retrieved documents, and clicked document metadata. To ensure user privacy, Personally Identifiable Information (PII) was removed.
From this dataset, the authors construct the TripClick IR benchmark. This involves:
- Document Collection: A subset of click logs pointing to MEDLINE articles was selected, resulting in about 1.5 million medical article abstracts.
- Query Set: Approximately 692,000 unique queries were extracted. These queries were categorized into three frequency groups: HEAD (most frequent), TORSO (moderately frequent), and TAIL (rare).
- Relevance Signals: Two types of query-document relevance signals were generated using click-through models:
- RAW: A simple approach where any clicked document is considered relevant (score of 1). Non-clicked documents from the ranked list before the clicked one are considered non-relevant (score of 0).
- DCTR (Document Click-Through Rate): This model estimates relevance as the click rate for a document over all its retrieved instances for a given query. This is a more nuanced measure. DCTR relevance is discretized into four grades (0-3).
The authors also created a large training set using a pairwise learning-to-rank format, where each item consists of a query, a relevant document, and a randomly selected non-relevant document.
Datasets
- TripClick Logs Dataset: A newly released dataset of approximately 5.2 million user interactions from the Trip Database health web search engine (2013-2020).
- TripClick IR Benchmark: Derived from the TripClick Logs Dataset, this benchmark includes:
- A collection of ~1.5 million MEDLINE article abstracts.
- ~692,000 unique queries, categorized into HEAD, TORSO, and TAIL frequency groups.
- ~1.3 million query-document relevance signals (using RAW and DCTR models).
- Pre-defined training, validation, and test sets for model development and evaluation.
The paper also references other datasets for comparison, such as Sogou-QCL and MS MARCO, highlighting TripClick's unique scale and domain specificity.
Key Insights and Ideas
- Health Domain Data Scarcity: The paper strongly emphasizes the lack of large-scale, domain-specific click log data for health IR, which hinders the development of advanced models in this critical area.
- Neural IR Advantage: The experiments demonstrate that neural IR models significantly outperform classical IR models on the TripClick benchmark, especially for more frequent queries. This suggests that the rich signals in click logs are particularly beneficial for learning complex patterns captured by neural networks.
- Query Frequency Matters: The performance gains of neural models are more pronounced for HEAD and TORSO queries compared to TAIL queries. This is intuitive, as more frequent queries provide more robust signals for learning.
- COVID-19 Data Potential: The inclusion of data up to October 2020, and specifically the analysis of COVID-19 related queries, highlights the dataset's potential for studying real-time information needs and search engine adaptation during public health crises.
Key Results
The evaluation results, presented in Tables 4 and 5, show that neural IR models consistently outperform classical baselines (BM25, RM3 PRF) across various metrics (NDCG, MRR, Recall) and query groups (HEAD, TORSO, TAIL).
- The TK (Transformer-Kernel) model achieved the best overall performance, significantly outperforming classical models across all metrics and groups.
- For the HEAD group using RAW relevance, the TK model achieved an NDCG of 0.302, MRR of 0.521, and Recall of 0.174 on the test set.
- For the HEAD group using DCTR relevance, the TK model achieved an NDCG of 0.221, MRR of 0.453, and Recall of 0.194 on the test set.
- The paper notes that neural models generally show more significant improvements for more frequent queries (HEAD > TORSO > TAIL).
Limitations and Future Work
- Relevance Signal Limitations: The authors acknowledge that the RAW and DCTR relevance signals are derived from click-through models and may not perfectly capture true user relevance. They suggest that the dataset can be used to develop more advanced click-through models in the future.
- DCTR Applicability: The DCTR model is most meaningful for frequent queries (HEAD group) due to the need for sufficient data to estimate reliable click rates.
- Future Work: The authors envision future work utilizing the dataset for:
- Developing more sophisticated click-through models.
- Exploring list-wise learning-to-rank approaches, given the abundance of relevance signals.
- Further research into neural IR models specifically tailored for the health domain.
- Investigating user behavior and information needs during critical events like the COVID-19 pandemic.
Summary updated