Matching Structure for Dual Learning

Hao Fei, Shengqiong Wu, Yafeng Ren, Meishan ZhangICML 2022 · OpenReview

Many natural language processing and machine learning tasks naturally exist in dual forms. If a model can translate English to French, it stands to reason it should be able to translate French back to English. Standard dual learning exploits this symmetry by treating the two tasks as a closed loop, ensuring that translating a sentence and then translating it back yields something close to the original input.

However, current dual learning schemes often treat these tasks as "black boxes." They focus entirely on the final output while ignoring the rich, intermediate structural correspondences between them. For example, in translation, a specific noun phrase in the source sentence usually maps to a corresponding noun phrase in the target. In image captioning, specific words map to specific visual regions. When models ignore these internal "Regions of Interest" (RoIs), they miss out on a powerful learning signal.

Standard dual learning (left) versus the proposed structure-matching approach (right).

To address this, researchers have developed StruMatchDL, a framework that supercharges dual learning by explicitly matching the underlying structures of coupled tasks. By aligning syntactic trees in text or semantic regions in images, StruMatchDL significantly improves performance across 28 different tasks, ranging from machine translation to image synthesis.

The StruMatchDL Framework

The authors introduce two main components to the vanilla dual learning framework to enforce structural awareness:

The StruMatchDL framework featuring RoI Aligning and Structural Cross-Reconstruction.

1. Syntactic and Semantic RoI Alignment The model identifies structural components and forces them to align across modalities.

  • For text-to-text tasks (like translation), it uses constituency trees to identify phrases. An attention mechanism combined with contrastive learning "pushes" corresponding phrases from both tasks closer together in the latent space while repelling unrelated ones.
  • For text-to-non-text tasks (like image captioning), where one side lacks a formal syntax tree, the model aligns the text's syntactic structures with the non-text's semantic regions (e.g., object bounding boxes detected by a Faster R-CNN).

2. Structural Cross-Reconstruction During the generation process, the model doesn't just predict the target output; it also attempts to reconstruct the structural "skeleton" (e.g., the syntax tree) of the input from the opposite task. This provides a global constraint that ensures the generated output follows a logical, well-planned structure. To handle these hierarchical structures, the architecture utilizes TreeLSTMs integrated with standard backbones like Transformers or BART.

Key Insights and Results

The researchers evaluated StruMatchDL across 14 pairs of dual tasks (28 tasks in total), including Neural Machine Translation (WMT14), Paraphrase Generation (ParaNMT, Quora), and Text-to-Image synthesis (MSCOCO, Flickr30k). The results highlight several core advantages of structure matching:

  • Structure as a Universal Bridge: The fundamental insight is that structure—whether syntax in text or semantic regions in images—acts as a universal language bridging different modalities. Even if an image lacks "grammar," the spatial and semantic relationships of its objects mirror the grammar of a descriptive sentence.
  • Improved Content Planning: By forcing the model to align structures, it becomes better at "content planning." Instead of merely predicting the next word, the model maps out which parts of the input should be addressed at which stage of the output.
  • Significant Performance Gains: On WMT14 English-German translation, the method improved BLEU scores by up to +3.57 over vanilla dual learning. For text-to-image synthesis on MSCOCO, the Inception Score (IS) rose from 28.4 to 30.7, and the FID (where lower is better) dropped from 24.8 to 20.6.
  • Unsupervised Alignment: Remarkably, the model learns these structural alignments in an unsupervised manner during dual training, achieving over 85% accuracy compared to human-annotated "gold" alignments. Human evaluators also rated the generated text as significantly more grammatically correct and content-rich.

Limitations and Future Work

While highly effective, StruMatchDL currently relies on external parsers (like StanfordNLP) to generate initial syntax trees. If the parser introduces errors, that noise propagates into the dual learning process. Additionally, encoding and matching hierarchical trees via TreeLSTMs introduces computational overhead compared to standard sequence-to-sequence modeling.

Moving forward, the authors aim to explore unsupervised structure induction, allowing the model to discover the underlying structures of the data entirely on its own, without relying on external parsers or predefined grammar rules.

Summary updated

Abstract

Many natural language processing (NLP) tasks appear in dual forms, which are generally solved by dual learning technique that models the dualities between the coupled tasks. In this work, we propos...
BibTeX
@inproceedings{fei2022matching,
  author = {Hao Fei and Shengqiong Wu and Yafeng Ren and Meishan Zhang},
  title = {{Matching Structure for Dual Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=ap-pxOQjO1},
}
Figure