Abstract
Cross-language sentence similarity computation is among the focuses of research in natural language processing (NLP). At present, some researchers have introduced fine-grained word and character features to help models understand sentence meanings, but they do not consider coarse-grained prior knowledge at the sentence level. Even if two cross-linguistic sentence pairs have the same meaning, the sentence representations extracted by the baseline approach may have language-specific biases. Considering the above problems, in this paper, we construct a Chinese–Uyghur cross-lingual sentence similarity dataset and propose a method to compute cross-lingual sentence similarity by fusing multiple features. The method is based on the cross-lingual pretraining model XLM-RoBERTa and assists the model in similarity calculation by introducing two coarse-grained prior knowledge features, i.e., sentence sentiment and length features. At the same time, to eliminate possible language-specific biases in the vectors, we whitened the sentence vectors of different languages to ensure that they were all represented under the standard orthogonal basis. Considering that the combination of different vectors has different effects on the final performance of the model, we introduce different vector features for comparison experiments based on the basic feature splicing method. The results show that the absolute value feature of the difference between two vectors can reflect the similarity of two sentences well. The final F1 value of our method reaches 98.97%, which is 19.81% higher than that of the baseline.
Introduction
Determining semantic similarity between sentences in different languages is a fundamental challenge in NLP, with applications ranging from machine translation to plagiarism detection. However, cross-lingual similarity is notoriously difficult due to language-specific biases in vector space (anisotropy) and the neglect of "coarse-grained" sentence-level information like sentiment and structure. This paper addresses these issues specifically for the Chinese–Uyghur language pair, which lacks robust open-source datasets.
Proposed Methodology
Siamese XLM-RoBERTa Backbone
The core of the proposed method is a Siamese Network architecture utilizing XLM-RoBERTa. Sentences from both languages are fed into identical neural networks with shared weights to extract deep semantic embeddings ($U$ and $V$). To enhance these embeddings, the model incorporates two "prior knowledge" features: sentence sentiment (categorized as negative, neutral, or positive) and sentence length (character count).
Whitening and Feature Fusion
To solve the language bias problem, a whitening transformation is applied to the sentence vectors. This maps the vectors of both languages into a standard orthogonal basis, aligning their coordinate systems and zeroing out the mean. For classification, the researchers found that splicing the absolute difference between the vectors ($|U - V|$) alongside the original vectors was a far stronger indicator of similarity than simple concatenation, as it directly captures the semantic "distance."
Regularization via Flooding
Given the relatively small size of the custom dataset, the team employed a "flooding" method during training. This technique prevents the loss function from dropping to zero, forcing the model to continue exploring the parameter space and preventing overfitting on the training data.
Dataset Construction
As no open-source Chinese–Uyghur similarity dataset existed, the authors created a balanced dataset of 30,396 sentence pairs. They sourced 15,198 Tencent news headlines in Chinese, which were translated into Uyghur by researchers to create positive samples. Negative samples were then manually generated by inverting the semantics of the Uyghur translations.
Key Results and Insights
The proposed method achieved a dominant F1 score of 98.97%, representing a 19.81% improvement over BiLSTM-CNN baselines. Key findings include:
- Layer Selection: Vectors from the penultimate layer of XLM-RoBERTa outperformed the final layer, as they retain more general semantic information.
- Feature Impact: Adding sentiment and length features consistently improved performance, helping the model resolve ambiguities that fine-grained tokens might miss.
- Structural Understanding: The model demonstrated high accuracy (80%) in identifying grammatically incorrect sentences, suggesting a deep understanding of cross-lingual structural relationships.
Conclusion
By combining deep semantic embeddings with coarse-grained prior knowledge and applying whitening transformations, this method significantly advances the state-of-the-art for Chinese–Uyghur sentence similarity. Future work will focus on expanding the dataset and exploring additional features like syntactic tree depth and tense.
Summary updated