Abstract
The short text matching task employs a model to determine whether two short texts have the same semantic meaning or intent. Existing short text matching models usually rely on the content of short texts which are lack information or missing some key clues. Therefore, the short texts need external knowledge to complete their semantic meaning. To address this issue, we propose a new short text matching framework for introducing external knowledge to enhance the short text contextual representation. In detail, we apply a self-attention mechanism to enrich short text representation with external contexts. Experiments on two Chinese datasets and one English dataset demonstrate that our framework outperforms the state-of-the-art short text matching models.
Introduction
Short text matching is a fundamental task in Natural Language Processing (NLP), critical for applications like search engines, question-answering systems, and chatbots. The primary challenge lies in the "information-poor" nature of short texts. Brief snippets often lack the necessary context to resolve semantic ambiguity.
For instance, consider the sentences "How to calculate interest?" and "What is the interest rate normally?". While both likely refer to financial services, the specific domain (e.g., bank loans) is not explicitly stated. Without this shared context, a matching model might struggle to recognize their semantic similarity. Previous attempts to solve this using word-level knowledge (synonyms or dictionary definitions) often fall short of capturing the broader intent. This research proposes that sentence-level contextual information is required to bridge this semantic gap.
Methodology: Context-aware BERT Matching (CBM)
To address the lack of information in short texts, we introduce the Context-aware BERT Matching (CBM) framework. This model enhances short text representations by incorporating external context retrieved from search engine clickthrough data. The framework operates in three distinct stages:
1. Context Retrieval and Selection
The process begins with a Context Crawler that uses the original short sentences as search queries. The top results from search engine clickthrough data are retrieved as potential external contexts. To handle the inherent noise in search data, a Context Selector—a BERT-based module—scores the relevance of these retrieved contexts against the original sentences. Only the most semantically relevant contexts are retained for the next stage.
2. Context-Enhanced Encoding
The selected contexts and the original sentences are processed through a dual-encoder setup. As shown in the architecture diagram, the Context Encoder and Sentence Encoder (both BERT-based) generate initial embeddings. These embeddings are then concatenated to form a unified representation.
3. Transformer-based Fusion and Result Selection
The combined representation is fed into a 3-layer Transformer module. This module utilizes a multi-head self-attention mechanism to allow the model to "cross-pollinate" information between the short text and its external context, effectively enriching the semantic representation. Finally, a Result Selector combines the output of this context-enhanced path with a standard BERT baseline. This late-fusion approach ensures reliability: the model can rely on the baseline when the original text is clear or leverage the context-enhanced prediction when ambiguity is high.
Experimental Evaluation
The CBM framework was evaluated across three major benchmarks:
- BQ (Bank Question): A Chinese corpus of bank customer service logs.
- LCQMC (Large-scale Chinese Question Matching Corpus): A general-purpose Chinese intent matching dataset.
- QQP (Quora Question Paraphrasing): A standard English dataset for duplicate question detection.
Key Results
The proposed model consistently achieved state-of-the-art (SOTA) performance. On the BQ dataset, CBM reached an accuracy of 86.16% and an F1 score of 87.44%, notably outperforming previous models like Glyce+BERT. On the LCQMC dataset, utilizing a RoBERTa backbone further pushed the accuracy to 89.2%.
Ablation studies confirmed the importance of each component. Removing the context-sharing mechanism or the context selector led to significant performance drops, highlighting that filtering noisy search data and effectively fusing context are both critical to the model's success.
Conclusion and Future Work
This work demonstrates that sentence-level external context significantly improves short text matching by resolving semantic ambiguities that word-level methods cannot. While the CBM framework sets new benchmarks, challenges remain regarding the latency of real-time context retrieval and the dependency on search engine quality. Future research may explore multi-granularity approaches that combine sentence-level context with fine-grained word-level knowledge for even more robust semantic matching.
Summary updated