Feature-Rich Two-Stage Logistic Regression for Monolingual Alignment

Md. Arafat Sultan, Steven Bethard, Tamara SumnerEMNLP 2015 · DOI

Introduction

Monolingual alignment is the fundamental task of identifying semantic correspondences between words or phrases in two sentences within the same language. This process is critical for various high-level natural language processing (NLP) applications, including paraphrase detection, question answering, and the evaluation of machine translation systems. Unlike bilingual alignment, which often relies on large parallel corpora, monolingual alignment requires a deep understanding of context and semantic similarity to resolve ambiguities.

Example of monolingual alignment between two semantically similar sentences.

Problem Definition

The primary challenge in monolingual alignment lies in balancing local lexical similarity with global sentence structure. Words do not exist in isolation; they "compete" or "cooperate" for alignments. For instance, in the sentences "UK troops stormed a police station" and "British armor crashed into a jail," the word "troops" might locally match with "soldiers" elsewhere in the text, making it a better fit than "armor." Previous alignment models often struggled to integrate these interdependencies, leading to errors where local matches were prioritized over globally coherent structures.

Methodology: Two-Stage Logistic Regression

To address these challenges, we propose a feature-rich, two-stage logistic regression framework that separates local evidence gathering from global decision-making.

Stage 1: Local Assessment

The first stage calculates an alignment probability ($\phi_{ij}$) for every possible word pair $(i, j)$ based on local features. These features include:

  • Lexical Similarity: Utilizing neural word embeddings and the Paraphrase Database (PPDB) via a ridge regression model.
  • Contextual Features: Incorporating both dependency-based (syntax tree relations) and surface-form (neighboring words) similarities.
  • Word Typing: Training specialized models for different categories of word pairs (e.g., content words vs. function words) to capture category-specific alignment patterns.
The two-stage logistic regression framework for global alignment refinement.

Stage 2: Global Refinement

The second stage assigns final alignment labels by refining the Stage 1 probabilities through a global lens. This stage introduces features that account for:

  • Competition: Evaluating if a word has a stronger potential match elsewhere in the sentence.
  • Cooperation: Identifying if neighboring words provide mutual support for an alignment (e.g., components of a named entity).
  • Optimal Matching: Employing a max-weighted bipartite matching algorithm to determine the mathematically optimal set of one-to-one pairings across the entire sentence pair.

Evaluation and Results

The model was evaluated on the MSR Alignment Corpus and the Edinburgh++ Corpus. It achieved state-of-the-art results, reaching an $F_1$ score of 92.1% on the MSR corpus (a 4.8% error reduction) and 88.5% on Edinburgh++ (a 7.3% error reduction).

Extrinsic evaluation on Semantic Textual Similarity (STS) tasks further validated the approach, as the system outperformed the vast majority of participants in the 2013 *SEM STS competition. Ablation studies highlighted that contextual features are often more influential than purely lexical ones, and that the global refinement in Stage 2 is essential for maintaining high recall.

Conclusion

The two-stage logistic regression approach demonstrates that monolingual alignment benefits significantly from a hierarchical processing of features. By first assessing local similarity and then resolving global interdependencies, the model effectively handles the complexities of semantic context. Future work will focus on extending this framework to handle long-distance relationships and more complex many-to-many phrasal alignments.

Summary updated

BibTeX
@inproceedings{sultan2015feature,
  author = {Md. Arafat Sultan and Steven Bethard and Tamara Sumner},
  title = {{Feature-Rich Two-Stage Logistic Regression for Monolingual Alignment}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1111},
}
Figure