Teaching Character-Level Recurrent Readers to Flip Sentiment at Morpheme Boundaries

Sentiment models usually treat a word as an indivisible unit. Words like careless, unshareable, or the Bangla forms built with the prefix nir- and the suffix -hin (both roughly "without") carry their polarity inside the word: a small bound piece, an affix, reverses or weakens the meaning of the stem. A model that looks words up whole either has never seen such a word or has seen it too rarely to learn its polarity. Two lines of work almost meet here. Recurrent sentiment models, which read text one unit at a time while carrying a running memory, have been taught to flip their sentiment state when they reach a negation word such as "not". Character-level recurrent encoders read words letter by letter and can represent unseen words. No one has asked the character reader to do, inside the word, what the sentence reader already does across words.

We propose a within-word linguistic regularizer for character-level recurrent word encoders. While the encoder reads a word letter by letter, we attach a small sentiment readout to its hidden state after every character. A training penalty then requires that readout to change in a prescribed way when the reader crosses a morpheme boundary. Crossing a negating affix should push the readout through a learned "flip" transformation. Crossing an inflection or other non-affective morpheme should leave it nearly unchanged. Supervision comes only from short affix lists and from the noisy, rule-based decompositions that already exist. The claim is that this penalty teaches a reusable flip operation, and that this operation carries over to spellings, code-mixed forms, and affix variants that the rules themselves cannot recognize. That carry-over is what separates the method from feeding the rules' output to the model as a feature.

Core Claim

Linguistically Regularized LSTM for Sentiment Classification showed that adding loss terms on word-to-word hidden-state transitions helps a bidirectional LSTM. It used a negation term that applies a learned transformation when a negator appears and a non-sentiment term that keeps the state stable across neutral words. With these terms the bidirectional LSTM rose from 79.3% to 82.1% accuracy on the Movie Review (MR) dataset, with no phrase-level treebank needed. Every one of those terms is defined over whole-word tokens drawn from a word lexicon. We move the same idea one level down, to character-to-character transitions inside a word, and we tie it to morpheme boundaries.

The central hypothesis can be checked directly. A character encoder trained with this penalty should assign correct polarity to morphologically negated words whose affix or spelling never appeared in the regularizer's supervision. An encoder given the same rule output only as an input feature should not. If both do equally well, the regularizer adds nothing beyond the rules and the idea fails.

Mechanism

Each word is read by two character-level LSTMs, which are recurrent networks with gates that control what the running memory keeps. One reads left to right and one reads right to left. The direction matters for the morphology. A suffix like -less arrives after the stem for the left-to-right reader, so that reader can form a stem polarity and then flip it. A prefix like un- or nir- arrives after the stem only for the right-to-left reader. The prefix rule is therefore applied to the backward reader and the suffix rule to the forward reader.

For a reader with hidden state $h_i$ after character $i$, define a readout $s_i = \mathrm{softmax}(A h_i)$ over the classes negative, neutral, and positive, where $A$ is a shared matrix. At a boundary position $b$ marked by the weak annotator, let $s^-$ be the readout just before the morpheme and $s^+$ the readout just after it. The within-word penalty is

$$ \mathcal{L}_{\text{morph}} = \sum_{b \in \mathcal{N}} \mathrm{KL}\!\left(s^+_b \,\|\, \mathrm{softmax}(T_{m_b}\, A h^-_b)\right) + \sum_{b \in \mathcal{I}} \max\!\left(0,\ \mathrm{KL}(s^+_b \,\|\, s^-_b) - \epsilon\right). $$

In this penalty, $\mathcal{N}$ is the set of boundaries at negating or intensifying affixes, and $T_{m_b}$ is a learned transformation shared by all affixes of the same class, such as "privative". $\mathcal{I}$ is the set of boundaries at inflectional or non-affective morphemes, and $\epsilon$ is a tolerance margin. The second term targets the neutrality gap: it makes tense and case endings unable to create polarity on their own.

The two readers' final states form the word vector. A sentence-level bidirectional LSTM consumes these vectors and keeps the original word-level regularizers. The whole system is trained end to end on

$$ \mathcal{L} = \mathcal{L}_{\text{sent}} + \alpha\, \mathcal{L}_{\text{morph}} + \beta\, \mathcal{L}_{\text{LR}}, $$

where $\mathcal{L}_{\text{sent}}$ is the sentence cross-entropy, $\mathcal{L}_{\text{LR}}$ is the word-level regularizer, and $\alpha$ and $\beta$ are weights.

Because $T$ is shared across a whole affix class, the model learns one "privative" operation rather than one per affix. This sharing is the source of the expected transfer to unseen forms.

Boundary marks come from three cheap sources. The first is affix lists: the English privative prefixes and suffixes, and the Bangla nir-, -hin, and ni markers documented in BanglaBook. The second is the rule-based decomposition of The Good, the Bad, and the Unknown, which strips affixes recursively and tracks polarity reversals. The third, optionally, is a character segmenter in the style of Morphological Segmentation with Window LSTM Neural Networks, whose bidirectional window LSTM reaches 95.2 and 96.6 F1 on Hebrew and Arabic segmentation and stays strong with 25% of its training data. That paper supplies the segmentation method; we add the affective objective it lacks.

Relation to Prior Work

Several papers motivate this work by showing that negation needs explicit treatment in recurrent models. Improving Sentiment Analysis with Multi-task Learning of Negation trains a shared lower layer of a bidirectional LSTM to tag negation cues and scopes. It beats single-task training in 6 of 8 setups and helps with as few as 10 annotated sentences. SNNN adds attention weights for sentiment and negation words. Leveraging Multi-grained Sentiment Lexicon Information concatenates word-level lexicon category embeddings to the input. All of these work on whole words, so a negation hidden inside joyless never reaches them. The low data need reported by Barnes et al. suggests that weak, sparse boundary supervision can also be enough for our setting.

Other papers supply the character-level machinery, and they also show its limits. Subword-level Composition Functions for Learning Word Embeddings shows that character LSTMs compose derivational morphology far better than models without subword information: 57.8% on derivational analogies against 9.6% for Skip-Gram. Its encoders, however, are trained as static embedding generators with no sentiment signal. Sentiment Analysis of Dravidian Code Mixed Data feeds a character CNN into an LSTM for code-mixed text.

Some results constrain the design. In the Dravidian work, TF-IDF logistic regression reached 0.68 weighted F1 on Malayalam and beat both the word LSTM (0.63) and the character model. This is direct evidence that an unguided character recurrent model can lose to n-grams. Revisiting the Hierarchical Multiscale LSTM finds that boundaries discovered without supervision in multiscale LSTMs need not be linguistic at all, even when language-modeling quality is good. We read both results as arguments for supplying boundaries from outside rather than hoping they emerge.

Using Morphological and Syntactic Structures for Chinese Opinion Analysis shows that averaging morpheme polarities fails and that structure-specific composition is needed, for example sign multiplication in verb-object compounds. Our learned $T$ is a soft, trainable version of those hand-written composition functions.

Research Program

We will build the two-reader encoder with the within-word penalty and attach it to a sentence bidirectional LSTM that carries the word-level regularizers of the LR-LSTM. We will then study three questions.

The first is whether the learned flip transfers. We hold out entire affixes and spelling variants from $\mathcal{N}$ and test on words that contain them.

The second is whether the inflection term reduces neutral-versus-weak-polarity errors. This is the dominant error in rule-based tagging, which the Good/Bad/Unknown authors call the "greedy/lazy" error.

The third is whether the per-character readout trajectory is interpretable. A correctly regularized reader should show a visible sign change at the affix.

We run English (SST and MR), Bangla (BanglaBook), and romanized code-mixed Tamil and Malayalam (FIRE 2020). For the code-mixed data, the affix lists are seeded from native-script morphology and matched only approximately, so the noise in the boundary marks is part of what we test.

Baselines

The naive reference is a word-level bidirectional LSTM, the 79.3% MR baseline reported in the LR-LSTM paper, together with the same architecture given character encoders but no within-word penalty. The second of these is also the main ablation. It has identical parameters, data, and training budget, and differs only in $\alpha=0$, so it isolates the regularizer. A second ablation keeps the penalty but assigns affix classes at random to the boundaries. This controls for the possibility that any auxiliary loss on the character states helps.

The strongest neural competitor that shares our design philosophy is the Linguistically Regularized LSTM. It reports 82.1% accuracy on MR, and we re-run it with our tokenization and tuning budget. Beating it means a statistically reliable gain on MR or SST. More importantly, it means a clear gain on the subset of test sentences whose polarity depends on a morphologically negated word, where the method should help most.

On Bangla, the strongest reported system is Bangla-BERT large at 0.9331 weighted F1 (BanglaBook). Random forest on word n-grams reaches 0.9106 on the same data. We do not expect to beat the BERT model overall with a small recurrent model. Success instead means closing most of the gap from 0.9106 at a small fraction of the parameters, and beating BERT on the morphologically negated subset and on the neutral class.

On Malayalam code-mixed text, the strongest reported result is TF-IDF logistic regression at 0.68 weighted F1. On Tamil it is the word-embedding LSTM at 0.65 (Dravidian Code Mixed). Beating them means exceeding those numbers on the official test splits.

For isolated unseen-word polarity, the reference is the rule system of The Good, the Bad, and the Unknown: over 63% three-way and over 81% two-way accuracy on 2,000 infrequent English words. We compare against it on any portion of that word set we can obtain. Otherwise we compare on a matched set of rare SST words that have word-level labels.

Every system gets the same training data, the same number of hyperparameter trials, and five random seeds.

Why not just…?

Why not just use character n-gram TF-IDF with a linear classifier? This is a real threat. It already wins on Malayalam (0.68 weighted F1), and a linear model sees -less as a feature. What it cannot do is condition the affix on the stem. A feature for -less adds a fixed weight whether the stem is care or harm, and harmless is positive. The structure-dependent composition shown in the Chinese opinion work is exactly this case. We will run character 1–5-gram logistic regression on every dataset. We will report it separately on the negated-word subset, split by whether the stem is positive or negative. If it matches our method there, the recurrent machinery is unnecessary and we will say so.

Why not just feed FastText or subword-composition embeddings into the LSTM? The subword composition paper shows that summing n-grams is weak on derivational morphology. Even its better LSTM composers are trained without sentiment, so nothing forces their vectors to put careful and careless on opposite sides. In the Dravidian work, FastText with an LSTM tied the best Tamil system at 0.65 but lost on Malayalam. We include the subword-composition encoder, frozen and then fine-tuned, as a baseline. The held-out-affix test shows whether sentiment fine-tuning alone recovers the flip.

Why not just compute the rule-based polarity of each word and concatenate it as a feature? This is the most serious alternative. Zeng et al. report that word-level input concatenation of lexicon categories works well. SNNN reports the opposite, finding that such input features overfit. Where the rules fire, a feature probably captures most of the benefit. Our method can only win where the rules do not fire: on held-out affixes, romanized variants, and slang. The decisive experiment therefore trains both systems with the rules' coverage deliberately restricted, then tests on words outside that coverage. If the feature model matches ours there, we will drop the regularizer claim.

Why not just fine-tune a pretrained Transformer such as BERT? BERT already wins overall on BanglaBook. Its subword tokenization, though, splits words by frequency rather than morphology, so there is no guarantee that nir- or -hin becomes its own piece. We will measure fine-tuned BERT on the morphologically negated subset. If it shows no weakness there, our contribution reduces to an efficiency argument, and we will report it that way.

Decisive Checks

The pivotal experiment is the held-out-affix split. For each language, we remove one affix class member at a time from $\mathcal{N}$, for example -less while keeping un- and -free. We then measure word-polarity accuracy on held-out-affix words, using SST word-level labels for English and lexicon-derived labels for Bangla, alongside sentence accuracy on sentences containing those words. The method succeeds if it beats the rule-feature model, the unregularized character model, and character n-gram logistic regression on this split by a margin larger than the seed variance.

The second check measures neutral-versus-weak confusion with and without the inflection term, on SST three-way and BanglaBook neutral reviews. In BanglaBook, 77.57% of the unique words in neutral reviews also appear in positive ones, which makes this a hard test.

The third check is the readout trajectory: the fraction of held-out negated words whose per-character readout changes sign within two characters of the true boundary. This measure shows whether the model learned the operation or only a correlate of it.