TL;DR
This paper introduces "cross-encoder data annotation," a novel technique that leverages powerful but computationally expensive cross-encoder models to generate high-quality training data for more efficient bi-encoder models in product matching. This approach significantly improves bi-encoder performance, especially when human annotations are scarce or unavailable, by effectively filtering and refining training examples.
What Problem Does This Paper Address?
In e-commerce, accurately matching product listings from various sellers to a canonical product is crucial for managing catalogs, improving search, and preventing duplicates. Deep learning models, particularly bi-encoders and cross-encoders, have shown promise for this task. Bi-encoders are efficient for large-scale retrieval but can be less accurate. Cross-encoders are highly accurate but computationally prohibitive for matching millions of products. The challenge lies in building highly accurate bi-encoder models that can scale to massive e-commerce catalogs, especially when obtaining sufficient high-quality human-annotated data for training is a bottleneck.
Methods
The core contribution is cross-encoder data annotation. The authors propose using a pre-trained, accurate cross-encoder model to "annotate" potential training data for a bi-encoder. Here's how it works:
- Candidate Generation: A known, efficient bi-encoder is used to retrieve a set of candidate products for a given query product.
- Cross-Encoder Scoring: The query product and each candidate are then fed into a more powerful cross-encoder model, which assigns a similarity score.
- Data Filtering/Annotation: Pairs with high cross-encoder scores are treated as positive training examples for the bi-encoder. This effectively filters out noisy or less confident matches that a bi-encoder might struggle with.
- Bi-Encoder Training: A new bi-encoder is trained using these cross-encoder-annotated pairs.
The paper also explores two scenarios:
- When human annotations are available: The cross-encoder annotations are intersected with human annotations. This means only pairs that both the human annotator and the cross-encoder agree on are used, leading to a more focused and potentially higher-quality training set.
- When human annotations are NOT available: The cross-encoder annotations are used directly to train the bi-encoder, offering a semi-supervised or even unsupervised approach to generating training data.
Datasets
The experiments were conducted on an in-house e-commerce dataset containing 104 million products. Each product entry included a product ID, title, and description (in Japanese). For model training and evaluation, only the product titles were used to avoid issues with varying description quality. The dataset was split into training, development, and evaluation sets, with specific subsets of 110K products used for training the known bi-encoder, known cross-encoder, and the new bi-encoder, respectively.
Key Insights and Ideas
- Leveraging the Best of Both Worlds: The paper elegantly bridges the gap between the accuracy of cross-encoders and the scalability of bi-encoders by using the former as a sophisticated "teacher" to guide the latter's learning.
- Annotation as a Filter: The cross-encoder acts as a high-precision filter, identifying the most confident matches. This is crucial because training a bi-encoder on noisy data can degrade performance.
- Intersection Improves Quality: When human annotations are available, intersecting them with cross-encoder predictions leads to a smaller but more effective training set. This suggests that focusing on highly confident examples, even if fewer, can be more beneficial than using a larger, noisier dataset.
- "Hard Negative" Mining: By using the cross-encoder to identify high-scoring but potentially incorrect pairs (hard negatives), the bi-encoder can learn finer distinctions between similar-looking products (e.g., different colors or sizes).
- Reducing Reliance on Human Annotation: The ability to train effective bi-encoders without human labels is a significant step towards more scalable and cost-effective product matching systems.
Key Results
- No Human Annotation: The proposed cross-encoder data annotation improved absolute accuracy by 4% compared to a baseline that relied on simple retrieval-based training data.
- With Human Annotation: When intersecting human annotations with cross-encoder annotations, the accuracy improved by 2% absolute.
- Efficiency: The paper highlights the massive speed difference: cross-encoders can take 65 hours for tasks where bi-encoders take 5 seconds, underscoring why bi-encoders are preferred for production. The proposed method allows bi-encoders to achieve near cross-encoder accuracy with bi-encoder efficiency.
- Training Data Size: The intersection approach resulted in a smaller number of training queries (36,872) compared to human annotation (46,160) or cross-encoder annotation alone (42,186), yet achieved better results, reinforcing the idea that focused data is key.
Limitations and Future Work
- Computational Cost of Cross-Encoder: While the proposed method makes bi-encoders more efficient, the initial step of using the cross-encoder still incurs a computational cost. The paper doesn't detail the exact cost of this annotation phase.
- Class Imbalance: The paper notes that as new data is integrated, the updated cross-encoder model might have more negative pairs, leading to class imbalance. Future work could address this by setting thresholds or ensuring proper class balance during data updates.
- Continuous Improvement: The authors suggest integrating updated cross-encoder models into training data for continuous improvement, which could be explored further.
- Generalizability: While tested on an e-commerce dataset, further validation on other domains or product types could be beneficial.
Summary updated