TL;DR
This paper explores using BERT-based Bi-Encoders for multilingual news article similarity, finding that this approach outperforms traditional Cross-Encoders for this specific task. They investigate various configurations, ultimately achieving a competitive result by ensembling multiple Bi-Encoder models.
What Problem Does This Paper Address?
Accurately determining if two news articles discuss the same topic is crucial for applications like recommendation systems and related article discovery. This task becomes significantly more challenging when dealing with articles in multiple languages, as was the focus of SemEval-2022 Task 8. Traditional methods often struggle with this multilingual aspect, and the paper aims to find an effective solution using modern transformer-based models.
Methods
The core of the paper revolves around the Bi-Encoder architecture. Unlike Cross-Encoders, which process both articles simultaneously, Bi-Encoders process each article independently through a BERT model. This allows for more efficient encoding, especially for large datasets. The paper explores several key aspects of this Bi-Encoder approach:
- Pretrained Models: They experimented with several multilingual BERT variants, including
bert-base-multilingual-uncased,bert-base-multilingual-cased, andxlm-roberta-base. - Pooling Methods: To obtain a single vector representation for each article from BERT's output, they tested different pooling strategies, with the CLS token concatenation proving most effective.
- Translation: They investigated whether translating all articles to English before encoding would improve performance, but found it did not.
- Data Splitting and Max Length: The impact of cross-validation folds and the maximum token length fed into BERT were analyzed. They found that a larger number of folds improved performance, and surprisingly, a smaller max length did not lead to better results.
- Ensembling: To maximize performance, they combined the predictions of multiple trained Bi-Encoder models using a weighted average ensemble, with weights determined by validation performance.
Datasets
The primary dataset used is from SemEval-2022 Task 8, which consists of pairs of multilingual news articles with an "Overall" similarity score. The dataset includes both training and evaluation sets, with a notable number of language pairs appearing only in the evaluation set. This highlights the challenge of building models that generalize well to unseen language combinations.
Key Insights and Ideas
- Bi-Encoder's Advantage: Counterintuitively, for this specific multilingual news similarity task, the Bi-Encoder architecture outperformed the more commonly higher-performing Cross-Encoder. This suggests that the independent encoding of Bi-Encoders might be better suited for handling the diversity of multilingual inputs.
- CLS Token is King: The simple strategy of concatenating the last four CLS token representations from BERT proved to be the most effective pooling method, outperforming CNN, LSTM, and max-pooling.
- Translation Doesn't Always Help: The hypothesis that translating everything to English would simplify the task was not borne out. This suggests that the multilingual BERT models are capable of learning cross-lingual representations effectively on their own.
- Max Length Matters (but not as expected): While it's often assumed that shorter text inputs are better for similarity tasks, they found that a smaller max length actually led to worse performance. This implies that important contextual information might be present later in the articles.
Key Results
The paper reports Pearson's correlation as the evaluation metric.
- The best individual Bi-Encoder model achieved a Pearson's correlation of 0.7107 on the evaluation dataset (experiment 8).
- The weighted average ensemble of four models achieved the best overall result, reaching 0.7425 on the evaluation dataset.
- This ensemble score ranked 12th out of 32 teams in the SemEval-2022 Task 8 competition.
- The Cross-Encoder baseline achieved a Pearson's correlation of 0.6188, significantly lower than the best Bi-Encoder.
Limitations and Future Work
- Generalizability: The authors acknowledge that their findings are specific to this task and experimental setup and should not be overly generalized.
- Data Quality: They identified an issue where extracted article body text might differ from the actual content due to factors like advertisements or page layout, suggesting potential improvements in data preprocessing.
- Language Pair Performance: They observed significant performance differences across language pairs, with some pairs performing poorly. Focusing on these underperforming pairs in future work could lead to further improvements.
- Model Diversity: While the ensemble improved performance, the correlation between the models in the final submission was high. Exploring more diverse model architectures or training strategies could potentially yield even better results.
Summary updated