Ensemble Regression for Semantic Textual Similarity

32 concepts32 papers1 proposal

How do we determine if two sentences mean the same thing? To a human, semantic similarity is intuitive but multi-layered. Consider the sentences: "The dog chased the cat" and "An animal pursued a feline." They share zero words in common, yet their underlying meaning is nearly identical. Conversely, "The dog chased the cat" and "The cat chased the dog" share identical words but describe entirely different events.

Capturing these nuances requires looking at language through multiple lenses simultaneously: surface word overlap, syntactic structures, distributional embeddings, and formal logic. Because no single metric can capture all of these dimensions perfectly, a powerful paradigm has emerged: Ensemble Regression. Instead of searching for a single master metric, researchers treat individual similarity measures as features and use supervised machine learning to learn how to optimally combine them.


The Spectrum of Heterogeneous Features

To build a robust similarity engine, an ensemble system extracts a wide variety of linguistic signals. These features generally fall into four distinct tiers, spanning from surface strings to deep logical structures:

  1. Lexical and Surface-Level Overlap: These metrics measure how many characters, n-grams, or words are shared between two texts. For instance, traditional machine translation evaluation metrics like BLEU and METEOR can be repurposed to measure sentence similarity, as demonstrated by Barrón-Cedeño in UPC-CORE: What Can Machine Translation Evaluation Metrics and Wikipedia Do for Estimating Semantic Textual Similarity?.
  2. Syntactic and Structural Alignment: These features analyze grammatical relationships, such as dependency trees or predicate-argument structures. By determining "who did what to whom," systems can differentiate between active and passive voice or catch meaning shifts caused by word order changes. This structural layer is a core focus in UOW: Semantically Informed Text Similarity and ISCAS_NLP at SemEval-2016 Task 1: Sentence Similarity Based on Support Vector Regression using Multiple Features.
  3. Distributional and Knowledge-Based Semantics: To bridge the gap between different words with similar meanings (like pursued and chased), models leverage external knowledge bases like WordNet or corpus-based representations like Latent Semantic Analysis (LSA). As shown by Banea et al. in UNT: A Supervised Synergistic Approach to Semantic Text Similarity, combining structured lexical hierarchies with statistical text corpora provides a reliable semantic safety net. Similarly, Jha et al. demonstrate in Penn: Using Word Similarities to better Estimate Sentence Similarity that stacking complementary embedding spaces (neural, spectral, and context-based) captures a broader range of meaning than any single model alone.
  4. Deep Contextual and Logical Signals: Modern approaches incorporate dense vector spaces alongside formal logic. For example, Yanaka et al. in Determining Semantic Textual Similarity using Natural Deduction Proofs extract features from the internal mechanics of formal logic proofs—such as which sub-goals were skipped or which axioms were needed—balancing statistical patterns with mathematical precision. More recently, Günzler et al. in Sövereign at The Perspective Argument Retrieval Shared Task 2024: Using LLMs with Argument Mining integrated Large Language Model (LLM) relevance scores with demographic and topical features.

Why Regression Works: Learning the Fusion Function

Simply averaging these hundreds of features fails because their scales differ and their reliability depends heavily on the context. Supervised regression solves this by mapping a high-dimensional feature vector to a continuous, human-assigned similarity score (typically on a graded 0–5 scale, as standardized in the SEM 2013 shared task: Semantic Textual Similarity).

[ Sentence Pair ] 
       │
       ├──► Lexical Overlap (BLEU, n-grams) ──────┐
       ├──► Syntactic Alignment (Dependencies) ───┼──► [ Supervised Regressor ] ──► Final Similarity Score
       ├──► Semantic Vectors (Embeddings, LSA) ───┤          (e.g., SVR, RF)
       └──► Knowledge Graphs (WordNet paths) ─────┘

Different regression architectures offer unique advantages for this fusion task:

Linear and Log-Linear Regression

These models serve as highly interpretable aggregators. In DLS$@$CU: Sentence Similarity from Word Alignment and Semantic Vector Composition, Sultan et al. used linear regression to cleanly bridge discrete word-alignment counts with continuous vector similarities. On a larger scale, Bär et al. in UKP: Computing Semantic Textual Similarity by Combining Multiple Content Similarity Measures scaled this approach to over 300 heterogeneous features, proving that a log-linear model can effectively weight massive feature sets to ensure robustness across diverse text domains. Tran also demonstrates the efficiency of this approach using a streamlined linear model in TATO: Leveraging on Multiple Strategies for Semantic Textual Similarity.

Support Vector Regression (SVR)

When features interact non-linearly—for instance, when a syntactic feature is only meaningful if a certain threshold of lexical overlap is met—SVR models excel. Saric et al. in TakeLab: Systems for Measuring Semantic Text Similarity and Zhu and Lan in ECNUCS: Measuring Short Text Semantic Equivalence Using Multiple Similarity Measurements successfully deployed SVR to map complex combinations of dependency overlaps and latent semantic vectors into unified human-like scores. Buscaldi et al. in LIPN-CORE: Semantic Text Similarity using n-grams, WordNet, Syntactic Analysis, ESA and Information Retrieval based Features further extended this by treating sentences as search queries and feeding information retrieval metrics into an SVR framework.

Tree-Based and Stacked Ensembles

To handle noisy, high-dimensional data without overfitting, researchers turn to ensemble methods like Random Forests or multi-stage stacking. Alves et al. in ASAP: Automatic Semantic Alignment for Phrases introduced a stacked architecture where base learners (like K-Nearest Neighbors) pass their predictions to a meta-learner tree. In LIPN-IIMAS at SemEval-2016 Task 1: Random Forest Regression Experiments on Align-and-Differentiate and Word Embeddings penalizing strategies, Serrano et al. utilized Random Forests to dynamically weigh embedding alignments while penalizing words that appear in similar contexts but carry opposite meanings (such as "black" and "white").


Beyond Sentences: Broad Applications

The intuition behind regression-based similarity fusion extends far beyond standard sentence pairs. Because the framework is agnostic to the underlying feature types, it adapts naturally to complex downstream tasks:

By framing similarity not as a single mathematical calculation but as a supervised synthesis of diverse linguistic perspectives, ensemble regression provides a flexible, robust framework that closely mirrors the multi-faceted way humans understand meaning.

Key Foundational Work

All in Strings: a Powerful String-based Automatic MT Evaluation Metric with Multiple Granularities (2010) — contributed 1 concept elements to this direction.
A Study on Linking Wikipedia Categories to Wordnet Synsets using Text Similarity (2009) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_ensemble_similarity_regression_3657cd04,
  title = {ensemble similarity regression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/ensemble-similarity-regression/rdr_3657cd0482ca47d891bd8121e2d9303b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_ensemble_similarity_regression_3657cd04_v1,
  title = {ensemble similarity regression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/ensemble-similarity-regression/rdr_3657cd0482ca47d891bd8121e2d9303b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake