DeepSPIN: Deep Structured Prediction for Natural Language Processing

André Filipe Torres Martins

André F. T. Martins Proceedings of the 22nd Annual Conference of the European Association for Machine Translation (2020)

Abstract

The DeepSPIN project addresses the fundamental limitations of modern neural network models in Natural Language Processing (NLP) by integrating structured prediction techniques. By moving beyond "black-box" architectures, the project introduces sparse attention mechanisms and novel generation strategies to make machine translation (MT) systems more interpretable, data-efficient, and robust against common errors such as word dropping and repetition.

Introduction

While Transformer-based models have become the gold standard for NLP, they often rely on greedy search—predicting words sequentially without a global view of linguistic structure. This approach frequently results in "hallucinations," where models omit words or mistranslate named entities. Furthermore, standard attention mechanisms are "dense," assigning non-zero weights to nearly every word in a sentence, which obscures the model's decision-making process. The DeepSPIN project proposes a shift toward sparsity and structured prediction to resolve these issues.

Technical Innovations

Sparse Sequence-to-Sequence Models

Traditional models use the softmax function, which forces a non-zero probability on every word in the vocabulary. DeepSPIN utilizes entmax losses to create sparse probability distributions. This allows the model to focus exclusively on a small set of likely candidates, effectively "auto-completing" phrases when confidence is high.

Sparse generation and alignment: The left panel illustrates sparse probability distributions for word selection (auto-completion), while the right panel shows a clean alignment matrix between German and English.

Sparse and Structured Attention

By replacing dense attention with sparse variants like sparsemax, the model generates cleaner, human-readable word alignments. Additionally, the project incorporates "constrained" attention to model fertility. This ensures that source words are neither ignored (preventing omissions) nor translated multiple times (preventing repetitions).

Scheduled Sampling for Transformers

To mitigate "exposure bias"—where models fail when encountering their own errors during inference—the authors developed a new scheduled sampling technique tailored for the Transformer architecture, enhancing the robustness of the generation process.

Learning from the Translation Process

A key pillar of DeepSPIN is leveraging human post-editing data. Rather than just looking at the final translated product, the project analyzes keystroke sequences. This data provides insight into the cognitive effort required to fix a translation, allowing models to predict translation quality more accurately through weak supervision without needing massive new parallel corpora.

Key Results and Impact

  • Interpretability: Sparse attention acts as a built-in alignment tool, making the model's internal logic visible to human editors. Unlike blurry dense attention, sparse attention highlights specific source-target word relationships.
  • Reduced Errors: The use of constrained sparse attention significantly decreased word repetition in neural MT outputs by explicitly modeling word fertility.
  • Open-Source Contributions: The project released the OpenKiwi toolkit for translation quality estimation and the entmax library for PyTorch, both of which have become valuable resources for the NLP community.
  • Efficiency: The 1.5-entmax loss proved effective in providing manageable choice sets for next-word generation, facilitating interactive translation features.

Conclusion and Future Directions

DeepSPIN demonstrates that sparsity is a powerful feature for improving the reliability and transparency of NLP models. Future work will explore non-sequential generation strategies inspired by sequence tagging and refine hierarchical sparse attention for document-level translation, where long-range context is essential. The project continues to focus on reducing data dependency through advanced weak supervision techniques.

Summary updated

Abstract

André F. T. Martins. Proceedings of the 22nd Annual Conference of the European Association for Machine Translation. 2020.
BibTeX
@inproceedings{martins2020deepspin,
  author = {André Filipe Torres Martins},
  title = {{DeepSPIN: Deep Structured Prediction for Natural Language Processing}},
  booktitle = {EAMT},
  year = {2020},
  url = {https://openreview.net/forum?id=By_4kczTmr_},
}
Figure