When a text model trained on one domain, such as news, is applied to another, such as scientific papers, its accuracy drops. Adversarial domain adaptation is the standard fix. A second network, the domain discriminator, tries to tell which domain a sentence's internal representation came from. The main encoder is trained to fool it, so that source and target representations become indistinguishable. This works poorly when the two domains need different answers. It is provably harmful when the frequency of each label differs between domains: forcing the overall feature distributions to match then pushes target examples toward the wrong labels. Relation extraction is exactly this case. In relation extraction the model reads a sentence with two marked entities and names the relation between them. In the CrossRE benchmark, news text is dominated by one relation type and AI papers by another. Meanwhile, a separate line of work keeps finding that large pre-trained encoders already contain the grammatical structure that explicit parsers would supply. Explicit parse trees mostly add nothing, or add noise.
This note proposes to use that internal grammar as the variable to condition alignment on. We first group entity pairs into "syntactic configurations" by clustering the frozen pre-trained encoder's own representations, with no parser and no labels. We then run adversarial alignment separately inside each configuration, and reweight the configurations so their frequencies match across domains. The bet is that "how the two entities are grammatically connected" is stable across domains, observable without labels, and correlated with relation type. If so, it can stand in for the unavailable target labels. Matching distributions conditional on that stand-in, rather than globally, should avoid the failure that global matching runs into.
Core Claim
The theory and evidence motivating this come from three directions. On Learning Invariant Representation for Domain Adaptation proves that when label distributions differ, driving the source and target feature distributions to equality raises a lower bound on combined error. The same paper documents a "negative slope": target accuracy rises, then falls, as domain confusion becomes complete. The authors point toward conditional alignment as the remedy. Unsupervised Energy-based Adversarial Domain Adaptation for Cross-domain Text Classification observes the text version of the problem: standard adversarial adaptation on RoBERTa produces features that fool the discriminator but lose task information. Their fix, an autoencoder-based soft discriminator called EADA, still aligns a single pooled sentence vector. It never asks what linguistic information survives.
The claim is narrow and falsifiable. Unsupervised syntactic clusters taken from the frozen encoder are a better conditioning variable for adversarial alignment in cross-domain relation extraction than three alternatives: the model's own predicted labels, random partitions, or labels from an external parser. The benefit should grow with the amount of label shift between the source and target domain.
The closest existing work differs on the axis that matters. Silver Syntax Pre-training for Cross-Domain Relation Extraction shares the intuition that the grammatical path between two entities carries over across domains. It acts on that intuition by running an external parser over unlabeled text and pre-training on the predicted dependency labels. There is no alignment, and the method depends on parser quality. Syntax-Guided Domain Adaptation for Aspect-based Sentiment Analysis likewise uses external part-of-speech and dependency features, combined with hand-set similarity thresholds, to build pseudo-examples and choose which words to mask. We use neither a parser nor thresholds. The syntax comes from the encoder itself, and it enters only as the variable that decides which source and target examples are compared by the discriminator.
Research Program
The method has three components.
The first is the syntactic partition. For each entity pair we form a vector from the frozen pre-trained encoder at an upper-middle layer: the two entity-start token states concatenated with the mean of the tokens between them. Following Deep Clustering of Text Representations for Supervision-Free Probing of Syntax, we reuse their SyntDEC procedure, which compresses representations with an autoencoder and sharpens clusters with a divergence loss. That paper shows upper layers of transformers organize syntax into clusters that can be recovered without labels, and that clusters learned in English transfer to French and Japanese at 73.9% mean accuracy. We cluster source and target pairs jointly into $K$ groups. The partition is fixed before adaptation begins, so it cannot drift as the encoder moves. One risk is that clusters built from raw representations will capture vocabulary rather than grammar. To reduce it, the SyntDEC input uses the between-token and boundary states, not the entity tokens' word identity alone. We then check, as described below, how well the clusters predict domain versus how well they predict gold dependency paths.
The second component is conditional alignment. Let $h$ be the adapting encoder's entity-pair representation and $c \in \{1,\dots,K\}$ its fixed cluster. Let $D_c$ be a small domain discriminator for cluster $c$, connected to the encoder through gradient reversal, the layer that flips the discriminator's gradient so the encoder learns to hide domain identity. Let $w_c = p_T(c)/p_S(c)$ be the ratio of target to source cluster frequency, estimated from unlabeled data. Let $\mathcal{L}_{\text{task}}$ be the source relation-classification loss and $\lambda$ a weight on the alignment terms. Training minimizes
$$\mathcal{L} = \mathcal{L}_{\text{task}} + \lambda \sum_{c=1}^{K} \mathcal{L}_{\text{adv}}\big(D_c;\ \{h : c(h)=c\},\ w_c\big),$$
where source examples inside each cluster's adversarial loss are weighted by $w_c$. This weighting is the key step. Configuration frequencies are allowed to differ across domains, so the method never forces the global marginal match that Zhao et al. show is harmful. Only the representation within a configuration is pushed to be domain-invariant. To the extent that relation-label shift is carried by shifts in configuration frequency, the reweighting absorbs it.
The third component is a diagnostic study, which is useful whether or not the method wins. Throughout adaptation we track two things on held-out Universal Dependencies sentences: how much syntax survives in the adapting encoder, and how close its representations remain to the original pre-trained ones. Syntax survival is measured by SyntDEC part-of-speech many-to-one accuracy. Closeness is measured by linear centered kernel alignment, a similarity score between two sets of representations, used by Is Supervised Syntactic Parsing Beneficial for Language Understanding Tasks? An Empirical Investigation. The question is whether standard adversarial adaptation erodes the encoder's implicit syntax. We also ask whether the timing of that erosion predicts the negative slope in target relation F1. Nothing in the file reports this measurement.
Baselines
The naive reference point is source-only fine-tuning. A BERT relation classifier with entity markers is trained on the labeled source domain and evaluated directly on each target domain, as in CrossRE: A Cross-Domain Dataset for Relation Extraction. That paper reports an average Macro-F1 of 28.29 across domains for this baseline, with Micro-F1 of 75.96 on music and 40.10 on natural science. The file does not make clear that this average comes from the same news-to-target unsupervised protocol we will use. We therefore rerun the baseline under our protocol rather than borrow the number. Every adapted method must beat this rerun on average target Macro-F1.
The strongest existing method on this benchmark is silver syntax pre-training (Silver Syntax Pre-training for Cross-Domain Relation Extraction). Its reported result is an average improvement of 0.71 Macro-F1 over the BERT baseline on CrossRE, beating the baseline in 5 of 6 setups. That is a small margin, so we will run it with its parser on the same unlabeled source and target text we use. Beating it means a larger average gain than its rerun gain, holding across seeds, with confidence intervals reported.
The strongest adversarial competitors are of two kinds. The first is a global gradient-reversal discriminator on the entity-pair vector. The second is EADA adapted to pair representations. For EADA, the file reports 86.44% average accuracy across 12 Amazon review pairs, about 2.5 points over standard adversarial RoBERTa (Unsupervised Energy-based Adversarial Domain Adaptation for Cross-domain Text Classification). We also include conditional adversarial adaptation, which conditions the discriminator on the classifier's own predictions (Conditional Adversarial Domain Adaptation). This is the most direct competitor, because it also conditions alignment, just on a different variable.
Fairness rules apply to every method. All methods share the same encoder, the same labeled source data and unlabeled target data, the same number of training steps, and the same hyperparameter search budget, including $\lambda$ and discriminator size. Model selection uses a small labeled source-domain validation split. Because adversarial methods are unstable, we report the mean and spread over at least five seeds, and we report results both at the selected checkpoint and at the end of training. The end-of-training number is what exposes the negative slope.
Four ablations isolate the key ingredient. The first replaces syntactic clusters with random partitions of equal size. The second replaces them with lexical clusters: k-means on bag-of-words vectors, which should track domain rather than grammar. The third uses clusters defined by parser-derived shortest-path relation labels, the explicit-syntax version of our own method. The fourth removes the reweighting by setting $w_c = 1$. The proposal counts as supported only if implicit syntactic clusters beat random and lexical partitions, at least match parser clusters, and lose noticeably when reweighting is removed.
Why not just…?
Why not just continue masked-language-model pre-training on target text? This is a real threat. Glavaš and Vulić's control experiment showed that seeing more target text accounts for most transfer gains that had been attributed to syntax (Is Supervised Syntactic Parsing Beneficial for Language Understanding Tasks? An Empirical Investigation). Against this, silver syntax pre-training still added 0.71 Macro-F1 over plain BERT fine-tuning on CrossRE, which suggests text exposure alone does not close the gap. That paper's comparison was not against domain-adaptive pre-training, however. We will therefore run continued pre-training as a baseline at equal compute, and also run our method on top of it. If our method adds nothing over continued pre-training, the claim fails.
Why not just condition on predicted labels, as conditional adversarial adaptation does? Predicted labels are the obvious stand-in for unavailable target labels. Under strong label shift, though, the source classifier's target predictions are least reliable for exactly the relation types that shift most. Conditioning on those errors can lock them in. Syntactic clusters are computed from a frozen encoder and do not depend on classifier accuracy. The head-to-head comparison settles this. Our prediction is that our advantage concentrates on the domain pairs with the largest relation-label shift, which we measure as the divergence between label distributions using the CrossRE annotations, used only for analysis. If label-conditioned alignment wins uniformly, the syntactic variable adds nothing.
Why not just parse the text and use the trees directly? Do Syntax Trees Help Pre-trained Transformers Extract Information? found that gains from injecting trees into BERT with graph networks largely disappear when predicted rather than gold trees are used, and that deep fusion of predicted trees hurt relation extraction on TACRED. Parsers are also least reliable on unfamiliar domains, which is exactly where adaptation is needed. Still, the parser-cluster ablation gives explicit syntax its best chance in our setting, since there it only has to supply a discrete grouping, not a full graph. If parser clusters clearly beat implicit clusters, we will report that the conditioning idea holds but the "implicit" part does not.
Why not just keep the encoder close to its pre-trained weights, by freezing layers or adding a distance penalty, so adversarial training cannot destroy syntax? If the only problem were adversarial training erasing structure, this cheaper fix would suffice. Preserving representations does nothing about label shift, however, which is the failure mode Zhao et al. identify. We include a weight-distance penalty combined with global adversarial alignment as a baseline. Our method should beat it specifically on high label-shift pairs. The diagnostic study will also show directly whether syntax erosion or label shift better explains the negative slope.
Proposed Experiments
The primary benchmark is CrossRE in the unsupervised setting. We train on labeled news and adapt to each of politics, natural science, music, literature, and AI using only unlabeled target text, then repeat with politics as the source. The metric is target Macro-F1 averaged over pairs, and gold entity spans are given, matching prior CrossRE protocols. As a secondary sanity check of the diagnostic claim on sentence-level transfer, we run global adversarial adaptation and EADA on the 12 Amazon review domain pairs. We track syntax survival during training there too, to test whether erosion is specific to fine-grained tasks.
Three results would decide the question. First, the gain over global and label-conditioned alignment should correlate positively with label-shift size across domain pairs. A flat or negative correlation would refute the mechanism. Second, implicit syntactic clusters should beat random and lexical partitions of equal size. If random partitions work as well, any gain comes from finer-grained discriminators rather than syntax. We also report how well clusters predict domain identity; clusters that strongly encode domain would undermine the premise. Third, under global adversarial adaptation, the drop in SyntDEC part-of-speech accuracy and in alignment with the pre-trained encoder should precede the decline in target F1. Under syntax-conditioned alignment, both should stay flat. If syntax survives standard adversarial training intact, the diagnostic half of the proposal yields a clean negative result. In that case the method's benefit, if any, must be attributed to the label-shift reweighting alone, which the $w_c = 1$ ablation measures directly.
References
- On Learning Invariant Representation for Domain Adaptation
- Unsupervised Energy-based Adversarial Domain Adaptation for Cross-domain Text Classification
- Silver Syntax Pre-training for Cross-Domain Relation Extraction
- Syntax-Guided Domain Adaptation for Aspect-based Sentiment Analysis
- Deep Clustering of Text Representations for Supervision-Free Probing of Syntax
- Is Supervised Syntactic Parsing Beneficial for Language Understanding Tasks? An Empirical Investigation
- CrossRE: A Cross-Domain Dataset for Relation Extraction
- Conditional Adversarial Domain Adaptation
- Do Syntax Trees Help Pre-trained Transformers Extract Information?