A Narrow Lexical Channel for Topic-Robust User Embeddings

Systems that personalize language understanding, such as sarcasm detectors that know who is speaking or authorship models that recognize a writer across platforms, usually summarize each person as a single vector learned from their past posts. That vector mixes two things: how the person writes (sentence shapes, function words, punctuation habits) and what they tend to write about. When a user's topics change, a vector that has absorbed their topics points the model in the wrong direction. Large pretrained encoders are the clearest example. Under a test that flips which author writes on which topic, they fall to near chance, while a feature-engineered SVM does far better.

We propose building user vectors on a syntactic backbone and admitting word-level information only through a deliberately narrow, gated side channel. The width of that channel is an explicit dial. It sets both the model's capacity and how much topical vocabulary can reach the user vector. The central claim is that this one dial governs the trade-off between topic robustness and accuracy. Specifically, the gap between same-topic and flipped-topic accuracy should grow as the channel widens, and the best width should grow with the number of posts available per user. If the claim holds, practitioners get a principled way to set capacity for user models instead of choosing between a crude POS-only model and a topic-absorbing large encoder.

Core Claim

Existing work offers two extremes and one heuristic. At one extreme, discarding words entirely is topic-proof but blind to meaning. The Gated POS-Level Language Model for Authorship Verification feeds only part-of-speech sequences, with a decoder split into shared and author-specific parts, and gains over 5% AUC-ROC on cross-topic Gutenberg. At the other extreme, combining syntax and words freely recovers accuracy but has never been tested under topic shift. Syntax Encoding with Application in Authorship Attribution concatenates a syntax-path channel with a character channel and reaches 96.16% on IMDB62, against 83.48% for syntax alone. Between them sits a heuristic: Topic-Regularized Authorship Representation Learning uses a TF-IDF "bias model" to discount topical shortcuts during distillation, gains on cross-topic Reddit and Fanfiction, and loses some in-distribution accuracy.

None of these treats the amount of lexical information as a controlled, measurable quantity. Our claim is that it should be one. A cross-attention layer can do this. Cross-attention is a mechanism where positions in one stream ask questions (queries) of another stream and receive weighted summaries of it (values). Here the syntactic stream asks and the word stream answers, and the answers pass through a rank-$r$ projection and a learned gate. Because every topical signal that reaches the user vector must pass through this rank-$r$ value path, $r$ becomes an upper bound on lexical leakage.

There is a subtlety that rules out the obvious design. Ordinary syntax-as-query cross-attention, in the style of the Hierarchical Multimodal Transformer to Summarize Videos, does not by itself block topic: the query only chooses positions, while the full-width values still carry topic words. The bottleneck is therefore the essential ingredient, and the attention and gate are secondary.

Research Program

The model has two streams. The syntactic stream is the sequence of part-of-speech tags, with function words and punctuation kept as literal tokens. It deliberately uses only a tagger rather than a full parser, because the file notes that parsers break down on informal text. The lexical stream is the token embeddings from a small frozen encoder.

These streams are fused with the gated cross-attention of An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention. That paper reports a gate that keeps a smaller modality from being "washed out" by a larger one, and sparsemax attention that focuses on a few positions; with these it lowered Davis MSE from 0.438 to 0.393. We reuse the technique but change its role. There the gate protected the weaker modality from the stronger one. Here it limits what the stronger, topic-laden stream may contribute. For syntactic position $i$, the fused feature is

$$h_i = (1-g_i)\, s_i + g_i \, B_r\, \textstyle\sum_j a_{ij} v_j,$$

where $s_i$ is the syntactic feature, $v_j$ are lexical token features, $a_{ij}$ are sparsemax attention weights, $B_r$ is a projection of rank $r$, and $g_i \in [0,1]$ is a learned gate.

A post vector is a shallow pooling of the $h_i$. The user vector is the mean of the user's post vectors. Mean pooling is used on purpose, because Plepi et al. and the PoliticES system of Fernandez de Landa and Agerri found that averaging is a strong user summary. We train with a triplet loss, as in DeepStyle. Wherever topic or genre labels exist, negatives are drawn from other authors writing on the same topic, so that matching topics cannot solve the task.

The analysis is a sweep over $r$, from 0 (syntax only) to full width (no bottleneck), crossed with the number of posts per user (10, 50, and full history). Three quantities are measured. The first is accuracy under topic shift. The second is the same-topic minus flipped-topic gap. The third is topic leakage, measured by how well a linear probe recovers topic labels from user vectors. The complexity literature motivates the prediction about history length. Gandhe et al. describe a data threshold below which simpler models win. Huang et al. find that attention encoders beat an MLP on Yahoo! R3 (15,400 users) but lose on Coat (290 users). From this we predict that the best $r$ moves upward as histories lengthen. This turns the open "crossover threshold" question into a curve that can be measured.

Baselines

The naive reference point is a frozen pretrained encoder whose post embeddings are mean-pooled into a user vector. This is the topic-sponge regime. The Topic Confusion Task reports 50–60% accuracy for BERT and RoBERTa on the flipped Guardian split.

The strongest classical competitor on that split is the same paper's SVM with POS n-grams, stylometric features, and word n-grams, which reports 83.3% accuracy. We rerun it on our exact splits. Beating it means exceeding its flipped-topic accuracy on repeated author partitions with non-overlapping confidence intervals. With only 13 authors, single-split wins mean little.

For cross-genre transfer, the strongest system is SELMA, the 7B-parameter style encoder in CROSSNEWS, which reports about 78.5% Article–Tweet and about 85.8% Article–Article verification accuracy. We do not expect a far smaller model to beat it outright. Success here means coming within a few points at a small fraction of its inference cost. This model gets no tuning budget from us, since we use it as released.

ARR is rerun on the Reddit and Fanfiction cross-topic splits with its TF-IDF bias model. The file reports only relative gains (6.95% Recall@8 and 8.7% MRR over a log-loss baseline); no absolute reported result was found on Lacuna for ARR. Beating ARR means higher cross-topic Recall@8 with no larger in-distribution drop.

Syntax-CNN with plain concatenation serves as the ungated two-stream reference; on CCAT50 it reports 81.00%. Ablations isolate the key ingredient. The full-width ($r$ = full) version tests whether the bottleneck matters. A version with the gate removed tests the gate. A version with softmax in place of sparsemax tests sparsity. A version without same-topic negatives tests the training signal.

All trained models receive the same tagger, the same training authors, the same equal number of hyperparameter trials, and matched training steps.

Why not just…?

Why not just train a POS-only model, as the Gated POS-RNN does? That model is the $r=0$ end of our sweep, so the experiment answers this question directly. The evidence that it leaves accuracy on the table is the jump from 83.48% to 96.16% on IMDB62 when Syntax-CNN adds content. Sarcasm detection also depends on words: CUE-CNN reaches 86.72% with user vectors combined with text. If $r=0$ matches the best $r$ under topic shift, we conclude that the lexical channel buys only in-topic accuracy, and we report that result.

Why not just mask topical content words and concatenate the two streams? This is a serious threat, because masking is a hard bottleneck with no learned parameters. We include it as a baseline that keeps function words and replaces content words with their tags. Our design should win only if some lexical information is stylistic, such as rare idiosyncratic words or characteristic transitions, and a learned narrow channel can keep it while masking cannot. If masking ties our model across the sweep, the lesson is that the bottleneck matters and attention does not.

Why not just add same-topic hard negatives to an ordinary encoder? This also may work. The ablation that removes the architecture but keeps the training signal (a full-width model with same-topic negatives) settles it. If it closes the topic gap as well as the narrow channel does, the contribution reduces to a training recipe, and we would say so.

Why not just use SELMA or a TF-IDF penalty? SELMA remains a 7B-parameter cost for each user query. ARR depends on an external surface-frequency heuristic and gives up some in-distribution accuracy. Both are run as baselines above, so the comparison is empirical rather than argued.

Proposed Experiments

The primary test is the Guardian Topic Confusion split, using repeated author partitions. The cross-genre test is CROSSNEWS Gold, Article–Tweet, at 10, 50, and 100 posts per author. Open-set cross-topic retrieval uses Reddit and Fanfiction with Recall@8 and MRR. A check of whether the user vectors still help a semantic task uses the Twitter sarcasm corpus, plugging our user vectors into a text classifier in place of CUE-CNN's vectors and comparing against its reported 86.72%.

There are three decisive checks. First, the topic gap and the probe's topic leakage should rise monotonically with $r$; a flat curve falsifies the central claim. Second, the accuracy-maximizing $r$ under topic shift should increase with posts per user; if it does not, the complexity-crossover framing fails. Third, at the best $r$, the full model must beat both the content-masking concatenation baseline and the full-width model trained with hard negatives under topic shift. Otherwise the gated narrow channel is not the cause of the gain.