Abstract
Traditional neural document representation learning often treats hand-crafted domain features as "side information," simply concatenating them to final layers. This paper proposes a feature-guided training method that maps feature indicators directly into the document's embedding space. By calculating similarity scores between word embeddings and feature indicators, the model gains a grounded understanding of how specific textual elements relate to target variables. Experiments on Wikipedia quality assessment and petition popularity prediction demonstrate that this approach consistently outperforms vanilla and side-information baselines across various architectures.
Introduction
In the current landscape of deep learning, automated representation learning has largely superseded manual feature engineering. However, domain-specific knowledge—such as the presence of specific keywords, structural markers, or linguistic patterns—remains highly valuable. When researchers attempt to integrate these hand-crafted features into neural networks, they typically use a "side-information" approach: concatenating numerical feature counts to the network's final hidden layers.
The authors of this study argue that this method is fundamentally "blind." The network receives a vector of numbers without context, lacking an inherent understanding of what these numbers represent or which parts of the input text they are derived from. This disconnect limits the model's ability to learn nuanced relationships between raw text and the target task. To bridge this gap, this paper introduces a feature-guided neural training method that explicitly grounds features within the document's content.
Methodology: Feature-Guided Training
The core innovation lies in how features are introduced to the model. Instead of treating features as external metadata, the proposed method integrates them into the representation learning process itself.
1. Feature Indicators and Shared Embeddings
The process begins by identifying "indicators" for each hand-crafted feature. For instance, if a feature tracks "level 3+ headings," the indicators might be the tokens === and ====. Both the document words and these feature indicators are projected into a shared embedding space.
2. Guidance Mechanism
For every word in a sentence, the model calculates the cosine similarity between the word's embedding and the feature indicator's embedding. These similarity scores are aggregated to create a "sentence score" for each specific feature. This score is then concatenated with the hidden representations learned by the neural network (e.g., a bi-LSTM or CNN).
By using this mechanism, the network is explicitly "guided" to recognize which parts of the text are semantically or structurally related to the important hand-crafted features.
Experimental Setup
The authors evaluated the feature-guided approach on two distinct supervised document representation tasks:
- Wikipedia Document Quality: A classification task involving 29,794 English Wikipedia articles labeled across six quality classes (from "Stub" to "Featured Article"). Features included reference counts, link density, and heading structures.
- Online Petition Popularity: A regression task using 10,950 UK petitions to predict the number of signatures. Features included the use of specific pronouns, sentiment-laden words, and biased language.
Results and Discussion
The feature-guided approach consistently outperformed both the "Vanilla" neural models and the "Side-information" baselines across multiple architectures, including CNNs, stacked bi-LSTMs, and models with hierarchical attention.
Performance on Wikipedia Quality Assessment
On the Wikipedia task, the feature-guided method showed statistically significant improvements. For example, using an Average bi-LSTM architecture, accuracy improved from 57.91% (Vanilla) to 59.90% (Feature-guided).
Performance on Petition Popularity
In the petition regression task, the feature-guided model achieved the lowest Mean Absolute Error (MAE) of 1.42 and Mean Absolute Percentage Error (MAPE) of 36.2. Notably, the standard "side-information" approach actually performed worse than the vanilla model in this instance, likely due to overfitting on raw feature counts. The feature-guided approach avoided this by focusing on semantic relationships rather than just numerical frequency.
Key Insights
- Semantic Expansion: The model can identify words semantically related to indicators even if they weren't in the original feature list. For example, the word "increase" showed high similarity to positive indicators like "help," effectively expanding the feature set automatically.
- Structural Awareness: By attending to structural tokens like
===, the model learns to value document organization, a critical signal for quality.
Conclusion and Future Work
Feature-guided training provides a more effective way to leverage domain expertise in neural networks. By grounding features in the embedding space, models achieve better performance and robustness.
One current limitation is compatibility with contextualized embeddings (e.g., BERT or ELMo). Since the proposed method relies on comparing a fixed feature indicator embedding to word embeddings, adapting it for dynamic, context-dependent encoders is a key area for future research. Additionally, the authors suggest that external signals, such as editor revision histories, could serve as valuable feature indicators for future document quality models.
Summary updated