TL;DR
This paper tackles the challenge of improving search result relevance in Azure Cognitive Search (ACS) for specific use cases like product search. They propose a novel method using Bayesian optimization to fine-tune ACS's internal weights, leading to significant improvements in key metrics like click-through rates on major Microsoft marketplaces.
What Problem Does This Paper Address?
Azure Cognitive Search (ACS) is a powerful "Search as a Service" cloud product, but a major hurdle for its users is tailoring its search relevance to their specific needs. Out-of-the-box configurations often result in suboptimal search experiences, leaving users unable to find what they're looking for. This is particularly critical for large-scale platforms like Microsoft's AppSource and Azure portals, which rely heavily on effective search to drive user engagement and business growth. The paper aims to bridge this gap by providing a data-driven approach to optimize ACS configurations for enhanced search relevance.
Methods
The core of the proposed solution lies in treating the ACS configuration as a black-box function that needs hyperparameter tuning. Specifically, the authors focus on optimizing the weights assigned to different searchable fields (like title, description, categories) and boosting features (like popularity, pricing type) within ACS. They leverage Bayesian optimization, a technique well-suited for optimizing non-differentiable systems where gradients are not readily available. The process involves:
- Collecting historical user interaction data: This includes click-through rates (CTR) for specific search queries and items.
- Defining an objective function: This function, based on normalized Discounted Cumulative Gain (nDCG), quantifies the quality of search results.
- Using Bayesian optimization (specifically Optuna): This framework intelligently explores the parameter space of ACS weights to find the combination that maximizes the nDCG score. The final objective function is weighted by search query frequency to prioritize improvements for popular terms.
Datasets
The paper utilizes historical user telemetry data from two major Microsoft portals:
- AppSource: A web portal offering Microsoft and third-party SaaS solutions.
- Azure Portal (Ibiza): A portal for managing and deploying Azure resources.
This data, collected over specific periods, consists of search queries and the corresponding click-through rates on displayed items. The authors do not introduce a new benchmark dataset but rather leverage existing production data from these large-scale marketplaces.
Key Insights and Ideas
- Treating search relevance as hyperparameter tuning: This framing is crucial. By viewing ACS's internal weights as hyperparameters, the problem becomes amenable to powerful optimization techniques like Bayesian optimization.
- Data-driven optimization is key: Relying on historical user interaction data (CTR) provides a direct measure of search effectiveness, allowing for tailored optimizations rather than generic heuristics.
- Bayesian optimization excels in this domain: Its ability to efficiently navigate complex, non-differentiable search spaces makes it ideal for tuning ACS, where direct gradient computation is not feasible.
- Query frequency weighting matters: Prioritizing improvements for frequently searched terms ensures that the optimization efforts have the most significant impact on overall user experience and business metrics.
Key Results
The paper reports significant improvements in search relevance:
- nDCG Lift: A substantial increase in nDCG was observed, with a +9.7% improvement for AppSource and +1.7% for the Ibiza portal.
- Real-world CTR Improvement: Live A/B testing revealed an approximate 1.2% increase in CTR on the Ibiza portal, demonstrating the practical impact of the optimized weights.
- Qualitative Improvements: Examples are provided showing how the optimized weights correctly rank relevant items higher in search results (e.g., ranking "DataFactory" first for the query "factory").
Limitations and Future Work
- Limitations:
- The optimization is dataset-specific, meaning weights optimized for one portal might not be directly transferable to another without re-optimization.
- The current optimization focuses on specific fields and boosting features; other aspects of ACS might also influence relevance.
- Accurate time-based cross-validation is challenging due to the dynamic nature of ACS and external factors.
- Future Work:
- The authors suggest extending the definition of search relevance to include other business-critical aspects like "transactability," "recency," or "popularity." This could be achieved by incorporating more searchable/boosting fields into the optimization loop.
- Exploring other optimization techniques beyond Bayesian optimization, such as genetic algorithms or iterated racing, could also be beneficial.
Summary updated