Reflection-based Word Attribute Transfer

Yoichi Ishibashi, Katsuhito Sudoh, Koichiro Yoshino, Satoshi Nakamura

TL;DR

Researchers have developed a novel way to flip word attributes (like changing "king" to "queen") by treating word embeddings as points in space and reflecting them across a "mirror" hyperplane. Unlike traditional methods that require explicit knowledge of a word's current state, this reflection-based approach automatically identifies which words to change and which to leave alone, achieving over 99% stability for non-attribute words.

What Problem Does This Paper Address?

In the world of Natural Language Processing (NLP), we often use word embeddings—vector representations where words with similar meanings sit close together. A famous property of these spaces is their ability to handle analogies: $\vec{king} - \vec{man} + \vec{woman} \approx \vec{queen}$.

However, using this for "attribute transfer" (e.g., changing gender or plurality) is harder than it looks. To turn "king" into "queen," you need to know that "king" is male so you can add the "female" vector. If you accidentally apply that same math to a gender-neutral word like "person" or a function word like "the," you end up with garbage.

Current methods rely on massive, costly lookup tables to tell the system which words have which attributes. The authors tackle a fundamental challenge: How can we transfer attributes automatically without needing to know beforehand if a word possesses that attribute?

Methods

The authors propose Reflection-based Word Attribute Transfer. Instead of simple vector addition, they use a geometric reflection operation.

The Geometric Intuition

Imagine a mirror (a hyperplane) in the embedding space. If you reflect the vector for "man" across this mirror, it lands on "woman." If you reflect it again, it lands back on "man." Crucially, if a word is neutral (like "person"), it should lie exactly on the mirror, meaning the reflection does nothing.

Figure 1: Examples of word attribute transfer

Parameterized Mirrors (REF+PM)

A single static mirror isn't enough because embedding spaces are often biased or non-linear. To solve this, the authors introduced Parameterized Mirrors. They use a Multi-Layer Perceptron (MLP) to dynamically generate the mirror's parameters—the normal vector $a$ and a point $c$—based on the specific input word and the target attribute.

Figure 3: Reflection with parameterized mirrors

This allows the model to be flexible: it can use slightly different mirrors for different neighborhoods of the word space (e.g., one mirror for royalty like king/queen and another for family terms like father/mother), as shown in the t-SNE visualization below.

Figure 6: Two-dimensional t-SNE projection of a

Datasets

The model was evaluated using 300-dimensional word2vec and GloVe embeddings across four binary attribute datasets:

  • Male-Female (MF): 53 word pairs (e.g., son/daughter).
  • Singular-Plural (SP): 140 word pairs (e.g., apple/apples).
  • Capital-Country (CC): 109 word pairs (e.g., London/UK).
  • Antonym (AN): 1,934 word pairs (e.g., hot/cold).
  • Non-attribute set ($N$): 1,000 words sampled from the vocabulary to test "stability."

Key Insights and Ideas

  • Involution is Key: An ideal transfer function should be its own inverse ($f(f(x)) = x$). Reflection naturally satisfies this property, whereas standard neural networks (MLPs) often struggle to "return" to the starting point accurately.
  • Distance as a Filter: The model learns to place non-attribute words on the mirror. The authors found that the distance between a word and its mirror is a reliable indicator of whether it carries the attribute. Neutral words have a distance near zero, while attribute-heavy words are distributed further away.
Figure 4: Distribution of distance between input word vector and its mirror |(vx−c)·a|
  • Local Context Matters: By parameterizing the mirror based on the input word, the model handles the inherent "noise" and bias in pre-trained embeddings much better than a single global transformation vector.

Key Results

The proposed REF+PM method was compared against standard MLPs and analogy-based difference vectors (DIFF).

  • Stability: This is the standout result. REF+PM achieved >99% stability across almost all datasets. This means it almost never incorrectly changes a neutral word. In contrast, a standard MLP baseline had stability as low as 1-5%.
  • Accuracy: REF+PM significantly outperformed analogy baselines that lacked explicit knowledge. For example, on the Capital-Country dataset using word2vec, REF+PM hit 58% accuracy, while the best analogy baseline without explicit knowledge only managed 32%.
  • Qualitative Success: In sentence-level tests, REF+PM successfully flipped "The woman got married when you were a boy" to "The man got married when you were a girl," while leaving "the," "got," "married," "when," "you," and "were" completely untouched.

Limitations and Future Work

  • Binary Focus: The current work is strictly limited to binary attributes (A $\leftrightarrow$ B). Extending this to multi-class attributes (e.g., color or tense) would require a more complex geometric approach than simple reflection.
  • Antonym Performance: While stability was perfect for antonyms, the raw transfer accuracy was slightly lower than a standard MLP (28.8% vs 35.9%). The authors suggest this is because the "antonym" relationship is much more semantically diverse and harder to capture with a single geometric rule.
  • Embedding Dependence: The method relies on the quality of the underlying pre-trained embeddings. If the initial space doesn't capture the attribute well, the mirror cannot fix it.

Summary updated

Abstract

Word embeddings, which often represent such analogic relations as king - man + woman = queen, can be used to change a word's attribute, including its gender. For transferring king into queen in this analogy-based manner, we subtract a difference vector man - woman based on the knowledge that king is male. However, developing such knowledge is very costly for words and attributes. In this work, we propose a novel method for word attribute transfer based on reflection mappings without such an analogy operation. Experimental results show that our proposed method can transfer the word attributes of the given words without changing the words that do not have the target attributes.
BibTeX
@inproceedings{ishibashi2020reflection,
  author = {Yoichi Ishibashi and Katsuhito Sudoh and Koichiro Yoshino and Satoshi Nakamura},
  title = {{Reflection-based Word Attribute Transfer}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure