Introduction
Unsupervised Domain Adaptation (UDA) remains a critical challenge in Natural Language Processing (NLP), aiming to deploy models on novel, unlabeled domains—such as social media or historical archives—without the prohibitive cost of manual annotation. Traditional approaches, notably Structural Correspondence Learning (SCL) and Denoising Autoencoders (mDA), typically rely on two limiting assumptions: the existence of manually selected "pivot features" that behave consistently across domains, and a simplified single-source to single-target adaptation scenario.
This paper introduces FEMA (Feature EMbeddings for domain Adaptation), a framework that moves beyond word-level representations to learn dense, structured embeddings for arbitrary NLP features. By treating representation learning as a feature-prediction task and decomposing embeddings into shared and attribute-specific components, FEMA provides a robust solution for multi-domain adaptation.
Methodology: Feature Template Embeddings
The core innovation of FEMA is the application of a Skip-gram-like objective to NLP feature templates rather than just words. Most NLP tasks utilize templates such as "current word," "previous word," or "suffix." FEMA learns embeddings for these specific features by training them to predict other active features within the same context.
Unlike word2vec, which ignores the syntactic role of a word, FEMA distinguishes between a word's role as the current token versus its role as a neighbor. This allows the model to capture finer-grained syntactic information, which is crucial for tasks like Part-of-Speech (POS) tagging.
Multi-Attribute Decomposition
To address the complexities of real-world data, where domains are often defined by multiple overlapping attributes (e.g., genre and time period), FEMA decomposes each feature embedding into a sum of components:
$$u_{feature} = h_{shared} + \sum z_{m} h_{attribute\_m}$$
Here, $h_{shared}$ represents a domain-invariant component, while $h_{attribute\_m}$ captures shifts specific to a particular domain attribute (indicated by $z_m$). By applying L2 regularization to the attribute-specific terms, the model is incentivized to store as much information as possible in the shared component, effectively "explaining away" domain-specific noise and isolating a robust global representation.
Experimental Evaluation
The authors evaluated FEMA on two challenging POS tagging tasks: the SANCL dataset (English web domains) and the Tycho Brahe Corpus (Portuguese historical texts).
Performance on Web Text
On the SANCL task, which involves adapting from the Wall Street Journal to diverse web domains like newsgroups and emails, FEMA demonstrated significant improvements over existing state-of-the-art methods.
As shown in the results, FEMA achieved an average accuracy of 91.71%, representing a roughly 10% error reduction over previous representation learning methods. Notably, FEMA maintains superior performance across a wide range of latent dimensions, proving more efficient and effective than SCL or mDA.
Historical and Multi-Genre Adaptation
The Portuguese Tycho Brahe Corpus provided a testbed for multi-attribute adaptation across four centuries and four genres. The multi-attribute version of FEMA consistently outperformed single-embedding variants. For instance, in the extreme case of adapting 19th-century models to 16th-century texts, the decomposition approach provided a 0.4% absolute accuracy boost, validating the utility of modeling specific domain attributes.
Conclusion
FEMA offers a flexible and computationally efficient approach to unsupervised domain adaptation. By shifting the focus from words to feature templates and explicitly modeling domain attributes, it achieves superior performance on syntactic tasks across diverse and evolving data sources. While the current implementation utilizes a classification-based approach (SVM) for efficiency, future work may explore the integration of FEMA embeddings into sequence labeling models like CRFs or LSTMs for more complex structured prediction tasks.
Summary updated