Some Like It Small: Czech Semantic Embedding Models for Industry Applications

Jirí Bednár, Jakub Náplava, Petra Barancíková, Ondrej LisickýAAAI 2024 · DOI

TL;DR

This paper introduces and evaluates "small" Czech sentence embedding models that achieve competitive performance with much larger models, making them ideal for real-time, resource-constrained applications. By leveraging techniques like multilingual distillation and unsupervised fine-tuning, these models significantly improve Seznam.cz's search functionalities, demonstrating the viability of efficient NLP for low-resource languages.

What Problem Does This Paper Address?

The field of Natural Language Processing (NLP) has seen tremendous progress thanks to sophisticated text representations like sentence embeddings. However, a significant trade-off exists between the size and computational efficiency of these models. Larger models often offer better semantic understanding but are too slow and resource-intensive for real-time applications. Furthermore, much of the research and development in this area has focused on English, leaving a gap for languages like Czech, which have limited labeled data. This paper aims to bridge this gap by developing and evaluating small, efficient Czech sentence embedding models suitable for industry applications, specifically within Seznam.cz.

Methods

The authors employed three primary methodologies to train effective small bi-encoder models:

  • RetroMAE: They utilized the retrieval-oriented pre-training method RetroMAE with a small BERT configuration. This involved an encoder and a decoder, with asymmetrical masking ratios applied during training.
  • Multilingual Distillation: This approach leverages a powerful English teacher model (all-mpnet-base-v2) to train a smaller Czech student model. They found initializing a new BERT model with a merged Czech-English vocabulary to be more effective than using a pre-trained multilingual model. They addressed the embedding size discrepancy between teacher and student by adding a linear projection and normalizing embeddings.
  • Unsupervised Fine-Tuning: All pre-trained models underwent further unsupervised fine-tuning using contrastive learning techniques. They experimented with SimCSE, RankCSE, and InfoCSE, using Wikipedia data for training. The core idea of SimCSE involves using a dropout mask as noise to create positive pairs, with in-batch negatives.

Datasets

  • Training Data:
    • Czech Corpus: A large, non-public corpus (253 GB) from Seznam.cz, comprising processed Czech web pages of varying quality.
  • czeng20-csmono: A high-quality, non-commercial English-Czech parallel corpus with 50 million sentence pairs.
  • Paracrawler v9: An open-source English-Czech parallel corpus with 50 million sentence pairs, suitable for commercial applications.
  • Czech Wikipedia dump: Used for unsupervised fine-tuning experiments.
  • Evaluation Datasets:
    • Intrinsic Evaluation:
      • Costra: Evaluates sentence embedding spaces for capturing linguistic phenomena like paraphrases and tense.
  • STS (Semantical Textual Similarity): Utilized three datasets: CNA (Czech journalistic texts), SVOB-IMG (image descriptions), and SVOB-HL (headlines).
  • Extrinsic Evaluation: * Multi-Label Classification (CTDC): A Czech text document corpus for document classification.
  • Sentiment Analysis (CFD): The Czech Facebook Dataset for sentiment analysis.
  • Relevance Ranking (DaReCzech): A dataset of Czech query-document pairs for information retrieval.

Key Insights and Ideas

  • Small is Powerful: The core finding is that small Czech models can achieve performance comparable to much larger "Base" models (approximately 8 times smaller and 5 times faster). This is crucial for real-time industry applications.
  • Multilingual Distillation Works: Training a Czech model by distilling knowledge from a strong English model is a highly effective strategy, especially when labeled Czech data is scarce.
  • Unsupervised Fine-Tuning Boosts Performance: Contrastive learning methods like SimCSE significantly improve the performance of pre-trained models, particularly in intrinsic evaluations and linear probing.
  • Autoencoders and Contrastive Learning Overlap: The study suggests a degree of interchangeability between autoencoder-based pre-training (like RetroMAE) and contrastive learning, as both lead to similar improvements in certain tasks.
  • Context Matters for Pooling: For models pre-trained with Masked Language Modeling (MLM), the choice of pooling method (e.g., MEAN, MAX) can significantly impact zero-shot performance, suggesting that optimal settings need to be determined empirically.

Key Results

  • Performance: The developed small Czech models (e.g., Dist-MPNet-ParaCrawl, Dist-MPNet-CzEng) achieved competitive results across various intrinsic and extrinsic evaluation tasks, often matching or exceeding larger models.
  • Efficiency: The small models are approximately 8 times smaller and 5 times faster than conventional Base models.
  • Seznam.cz Integration: These models have been successfully integrated into Seznam.cz's search engine, replacing older counterparts and leading to improved performance in organic search, featured snippets, and image search. For instance, replacing the Small-E-Czech text encoder with Dist-MPNet-ParaCrawl in image search led to a 7% increase in retrieving more relevant images.
  • Multilingual Distillation Advantage: The distilled models outperformed OpenAI's Ada embeddings on several tasks, despite being significantly smaller.

Limitations and Future Work

  • Data Scarcity: The primary limitation remains the scarcity of labeled Czech data, which necessitates the use of unsupervised and distillation techniques.
  • Model Replacement: While the new models have replaced older ones in production, the paper notes ongoing efforts to replace outdated models in earlier stages of the search pipeline.
  • Future Directions: The authors anticipate their released models will promote wider applications and inspire further research into efficient, smaller-scale models for complex language tasks. They also highlight the potential for adapting these techniques to other low-resource languages.

Summary updated

BibTeX
@inproceedings{bednar2024some,
  author = {Jirí Bednár and Jakub Náplava and Petra Barancíková and Ondrej Lisický},
  title = {{Some Like It Small: Czech Semantic Embedding Models for Industry Applications}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30307},
}
Figure