An Empirical Exploration of Local Ordering Pre-training for Structured Learning

Zhisong Zhang, Xiang Kong, Lori S. Levin, Eduard H. HovyEMNLP 2020 · OpenReview

Abstract

Recently, pre-training contextualized encoders with language model (LM) objectives has been shown an effective semi-supervised method for structured prediction. In this work, we empirically explore an alternative pre-training method for contextualized encoders. Instead of predicting words in LMs, we “mask out” and predict word order information, with a local ordering strategy and word-selecting objectives. With evaluations on three typical structured prediction tasks (dependency parsing, POS tagging, and NER) over four languages (English, Finnish, Czech, and Italian), we show that our method is consistently beneficial. We further conduct detailed error analysis, including one that examines a specific type of parsing error where the head is misidentified. The results show that pre-trained contextual encoders can bring improvements in a structured way, suggesting that they may be able to capture higher-order patterns and feature combinations from unlabeled data.

Introduction

Modern Natural Language Processing (NLP) relies heavily on pre-trained contextualized encoders. Most of these models, such as BERT, utilize Masked Language Modeling (MaskLM) as their primary pre-training objective. While MaskLM is highly effective at capturing lexical identity and semantic context, it may not explicitly focus on the structural scaffolding of language—specifically, the word order that conveys grammatical relationships.

Existing methods that incorporate word order often rely on global shuffling, which can be computationally intensive and creates a significant discrepancy between pre-training and downstream fine-tuning. This research explores a simpler, "local" ordering objective designed to help models capture structural patterns more effectively, particularly for tasks like dependency parsing and named entity recognition (NER).

Methods: Local Ordering Pre-training (LBag)

The proposed strategy, termed Local Ordering Pre-training (LBag), focuses on recovering local word order within a sentence. The process involves four key components:

  1. Local Bags of Words: Sentences are segmented into small "bags" (typically containing 3 to 7 words).
  2. Position Masking: Within each bag, all words are assigned an identical position index. This "blurs" the local order, forcing the encoder to rely on broader context to understand the sequence.
  3. Word-Selection Objective: Rather than using a complex decoder to reconstruct the entire sequence, the model employs a "word selector" (a biaffine scorer). The task is to identify the original neighbors (e.g., the word that was 1 or 2 positions to the left) for each word in the bag.
  4. Hybrid Training: A "Hybrid" model combines LBag with standard MaskLM, enabling the model to learn both lexical and structural patterns simultaneously.
Overview of the Local Ordering (LBag) pre-training objective.

This approach is computationally efficient and encourages the encoder to utilize the entire sentence context to resolve local structural ambiguities.

Experimental Setup

The models were pre-trained on 1 million Wikipedia sentences for each of four languages: English, Finnish, Czech, and Italian. Evaluation was conducted across three structured prediction tasks:

  • Dependency Parsing & POS Tagging: Using Universal Dependencies (UD) v2.4.
  • Named Entity Recognition (NER): Using CoNLL03 (English), Digitoday (Finnish), Czech Named Entity Corpus, and EVALITA 2009 (Italian).

To assess the impact of pre-training in data-constrained environments, the researchers tested "low-resource" scenarios by sampling only 1k, 5k, or 10k sentences for fine-tuning.

Key Results

The LBag method demonstrated consistent improvements over baselines without pre-training (Random and fastText) across all tasks and languages.

Performance comparison across parsing, tagging, and NER tasks in four languages.

As shown in the results, the performance gap is most pronounced in the 1k-sentence scenario. For instance, in English dependency parsing with 1k training sentences, LBag achieved a Labeled Attachment Score (LAS) of 82.25, significantly higher than the 69.93 achieved by the random baseline. While large-scale models like BERT-base still provide higher absolute performance due to their massive training corpora, the gap between the LBag-based models and BERT narrows as the amount of downstream training data increases.

Analysis and Insights

The study yielded several critical insights into how local ordering affects model learning:

  • Optimal Bag Size: Shuffling the entire sentence was found to be detrimental. The best results were achieved with "middle-ranged" bag sizes (around 7 words), suggesting that maintaining global structure while scrambling local details provides the most effective training signal.
  • Structural Improvements: Error analysis revealed that LBag helps models better identify complex structures like Multi-Word Expressions (MWEs) and conjunctions. In parsing, pre-trained models made fewer "head errors," and when errors did occur, they tended to be more "local" (attaching a word to a nearby neighbor) rather than "global."
  • Complementary Objectives: The Hybrid model consistently performed best, indicating that MaskLM (sensitive to lexical info) and LBag (sensitive to structural relationships) provide complementary benefits.

Conclusion

Local ordering pre-training offers a powerful and efficient alternative for enhancing contextualized encoders for structured prediction. By focusing on the "structural scaffolding" of language, LBag provides a significant "warm start" for models, particularly in low-resource settings. Future work may explore scaling this objective to larger models and investigating its interaction with sub-word tokenization.

Summary updated

Abstract

Recently, pre-training contextualized encoders with language model (LM) objectives has been shown an effective semi-supervised method for structured prediction. In this work, we empirically explore an alternative pre-training method for contextualized encoders. Instead of predicting words in LMs, we “mask out” and predict word order information, with a local ordering strategy and word-selecting objectives. With evaluations on three typical structured prediction tasks (dependency parsing, POS tagging, and NER) over four languages (English, Finnish, Czech, and Italian), we show that our method is consistently beneficial. We further conduct detailed error analysis, including one that examines a specific type of parsing error where the head is misidentified. The results show that pre-trained contextual encoders can bring improvements in a structured way, suggesting that they may be able to capture higher-order patterns and feature combinations from unlabeled data.
BibTeX
@inproceedings{zhang2020empirical,
  author = {Zhisong Zhang and Xiang Kong and Lori S. Levin and Eduard H. Hovy},
  title = {{An Empirical Exploration of Local Ordering Pre-training for Structured Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=x-ahMBfKKZK},
}
Figure