Composing Structure-Aware Batches for Pairwise Sentence Classification

Andreas Waldis, Tilman Beck, Iryna GurevychACL 2022 · DOI

Introduction

Pairwise sentence classification is a fundamental task in Natural Language Processing, encompassing applications such as paraphrase identification (QQP), natural language inference (NLI), and semantic textual similarity (STS). Traditionally, these tasks treat data points as independent pairs, feeding a model two sentences and a label in isolation. However, this approach ignores a critical structural property: many sentences appear in multiple different pairs across a dataset. This paper introduces "structure-aware batching," a method that organizes training data based on an underlying graph of sentence relationships to improve model performance.

The Annotation Graph

The authors observe that in many popular benchmarks, such as those in the GLUE suite, a single sentence often serves as a component in several pairs. By treating sentences as nodes and their annotated relationships as edges, one can construct an Annotation Graph (AG). Standard training typically shuffles these pairs randomly, effectively discarding the structural context provided by the graph. By grouping related pairs, the model can better "triangulate" the meaning of a sentence by seeing what it is—and isn't—related to in a single gradient update.

Construction of an Annotation Graph from individual sentence pairs.

Structure-Aware Batching Strategies

To leverage the AG, the researchers developed three batching strategies designed to present sentences to the model in multiple contexts within a single gradient update:

  1. NODE Strategy: Selects a set of unique sentences and includes all their associated pairs in the batch. This allows the model to compare a "hub" sentence against all its neighbors simultaneously.
  2. EDGE-I Strategy: Picks a "base" pair and samples "context" pairs from the immediate neighborhood of the two sentences. This strategy includes limits on neighborhood size to prevent highly connected nodes from dominating the batch.
  3. EDGE-II Strategy: A variation of EDGE-I that weights the loss calculation to account for varying node degrees, ensuring the model doesn't overfit to specific hub sentences.

These strategies were evaluated using both Bi-Encoders and Cross-Encoders across 14 heterogeneous tasks, including standard benchmarks like MNLI and QQP, as well as cross-topic argument mining tasks.

Key Results and Insights

The proposed strategies yielded significant improvements over standard random sampling. The EDGE-I strategy emerged as particularly robust, showing statistically significant improvements in 11 out of 14 tasks, with performance gains reaching up to 3.9%.

Performance distribution (F1) on QQP comparing the baseline against structure-aware batching strategies.

Beyond raw performance, the study yielded several key insights:

  • Low-Resource Efficiency: Structure-aware batching is exceptionally effective when training data is scarce. With as few as 25 instances, the structural information provides a significant performance boost over random sampling.
  • Graph Topology Matters: The effectiveness of a strategy depends on the graph's degree. In datasets with high-degree nodes, EDGE-I is superior as it prevents overfitting to specific hub sentences.
  • Model Agnostic: The benefits were consistent across various BERT-based architectures, including ALBERT and RoBERTa, suggesting the improvement is a fundamental property of the data-loading process rather than a model-specific quirk.

Conclusion

Structure-aware batching offers a simple yet effective way to improve pairwise sentence classification by acknowledging the inherent graph structure of datasets. While it requires overlapping sentences and a small amount of pre-processing, the resulting gains in performance and robustness—especially in low-resource settings—make it a valuable technique for modern NLP pipelines.

Summary updated

BibTeX
@inproceedings{waldis2022composing,
  author = {Andreas Waldis and Tilman Beck and Iryna Gurevych},
  title = {{Composing Structure-Aware Batches for Pairwise Sentence Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-acl.239},
}
Figure