Siamese BERT-Based Model for Web Search Relevance Ranking Evaluated on a New Czech Dataset

Matej Kocián, Jakub Náplava, Daniel Stancl, Vladimír KadlecAAAI 2022 · DOI

Abstract

Modern web search requires a delicate balance between semantic understanding and low latency. While Transformer-based Cross-Encoders offer high accuracy, their computational cost is prohibitive for real-time ranking. This paper details the development and deployment of a Siamese BERT-based model for the Czech language at Seznam.cz. By utilizing a Siamese Electra-small architecture with a custom neural interaction module and knowledge distillation, we achieved a 3.8% improvement in production relevance (P@10) with minimal latency overhead. We also introduce DaReCzech, a manually annotated dataset of 1.6 million Czech query-document pairs, to support future research in non-English information retrieval.

Introduction

The "gold standard" for text ranking is the Cross-Encoder approach, where a query and a document are fed into a model together. This allows for full self-attention between query and document tokens, providing high accuracy. However, this is computationally expensive at scale. Bi-Encoders (Siamese networks) allow for offline document embedding, enabling fast retrieval but often at the cost of accuracy due to limited interaction between query and document representations. Furthermore, Czech presents unique challenges as a morphologically rich language with limited open-source ranking resources.

Methods

Our approach bridges the gap between speed and accuracy through several architectural innovations. We utilize "Small-E-Czech," an Electra-small model pre-trained on a 253 GB Czech web corpus.

The Siamese architecture featuring the custom interaction module designed to capture nuanced query-document relationships beyond simple cosine similarity.

To enhance the Bi-Encoder's performance, we replaced standard cosine similarity with a custom interaction module. As shown in the architecture diagram, this module processes query and document embeddings through a 2-layer feed-forward network (FFN) incorporating element-wise maximums, residual connections, and Euclidean distances. To further close the performance gap, we employed knowledge distillation, using a high-accuracy Cross-Encoder as a teacher to provide soft labels for the Siamese student model.

Production Integration

The model is integrated into a multi-stage ranking pipeline to ensure both speed and relevance.

The multi-stage search ranking pipeline at Seznam.cz, where the Siamese model serves as a critical feature in Stage-2 ranking.

In production, the Siamese model's output is treated as one of over 500 features within a Gradient Boosted Regression Trees (GBRT) ranker. This allows the system to benefit from deep semantic features alongside traditional signals like BM25 and PageRank. The model specifically targets "Stage-2" ranking, processing approximately 20,000 documents down to the top 10.

Datasets

We introduce DaReCzech, a massive dataset of 1.6 million manually annotated Czech query-document pairs. This represents a significant contribution to non-English IR research. For evaluation, we utilized a "Train-big" set (1.4M records) for neural training and a "Train-small" set (97k records) for the GBRT ranker.

Results

The Siamese model achieved a P@10 of 45.26 on the DaReCzech benchmark, nearing the teacher Cross-Encoder's 46.30 while being approximately 1000x faster. In live traffic tests involving 3,000 queries, the model improved Precision@10 by 3.8% relative to the previous production baseline. Latency remains well within limits: query embedding generation takes ~10ms, and the ranking overhead is only ~3ms. Performance was further optimized using ONNX runtime with UINT8 quantization, yielding a 1.9x to 3.0x speedup with negligible loss in quality.

Conclusion

This work demonstrates that Siamese architectures, when augmented with custom interaction layers and trained via distillation, can provide Cross-Encoder-level relevance at production-scale speeds. The release of DaReCzech and Small-E-Czech provides essential infrastructure for Czech NLP and non-English information retrieval.

Summary updated

BibTeX
@inproceedings{kocian2022siamese,
  author = {Matej Kocián and Jakub Náplava and Daniel Stancl and Vladimír Kadlec},
  title = {{Siamese BERT-Based Model for Web Search Relevance Ranking Evaluated on a New Czech Dataset}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21502},
}
Figure