When humans are asked to generate text based on textual prompts—whether paraphrasing a sentence, describing a database key-value pair, or translating a phrase—they suffer from a strong cognitive bias known as lexical priming, lexical anchoring, or lexical entrainment. Instead of generating natural, varied language, human annotators subconsciously mirror the vocabulary, syntax, and structure of the prompt.
For instance, if a crowdsourcing prompt contains the attribute-value pair name: The Eagle, food: French, an annotator is highly likely to write "The Eagle serves French food," rather than a more organic phrasing like "If you're looking for a French meal, try The Eagle." This copying behavior results in robotic, repetitive datasets that lack the linguistic diversity of real-world human communication.
To break this linguistic lock-in, a growing body of research leverages visual elicitation. By replacing text-based prompts with non-linguistic visual stimuli—such as icons, diagrams, infographics, or short videos—researchers force the human brain to construct language from scratch, yielding significantly higher-quality training data for machine learning models.
Why Visual Stimuli Work: The Cognitive Shift
The core reason visual elicitation works lies in how the human brain processes information. A visual stimulus acts as a neutral semantic anchor. Because an image contains no words, it cannot prime the annotator's vocabulary or syntax.
[Text Prompt] --> (Cognitive Shortcut: Copy/Paste Tokens) --> [Stiff, Primed Text]
[Visual Prompt] --> (Deep Semantic Translation) --> [Natural, Diverse Text]
When presented with an image or an icon, the annotator cannot simply transcode tokens from the input to the output. Instead, they must perform a deep conceptual translation: internalizing the underlying meaning of the visual scene and then selecting words from their own mental lexicon to describe it. This cognitive shift bypasses text-to-text mirroring, encouraging the use of diverse synonyms, varied discourse markers, and organic sentence structures.
Breaking the Chains of Key-Value Pairs in NLG
The power of this approach is highly evident in Natural Language Generation (NLG). In pioneering work on crowdsourcing NLG data, Novikova et al. (2016) demonstrated that presenting crowd workers with pictorial representations (like icons arranged in a spatial layout) rather than text-based logic significantly reduced lexical priming. This simple shift forced participants to translate concepts into their own words, resulting in utterances that were more natural, informative, and better phrased.
This methodology laid the foundation for the E2E NLG Challenge. As detailed by Novikova et al. (2016) and later in the dataset's formal release by Novikova et al. (2017), using visual infographics instead of text-based key-value pairs forced human contributors to use diverse discourse markers and sophisticated groupings.
In the final evaluations and findings of the challenge, Dusek et al. (2019) and Novikova et al. (2018) confirmed that pictorial stimulus collection successfully prevented contributors from mimicking input structures. The resulting 50,000-instance corpus provided a massive, highly diverse benchmark that pushed neural sequence-to-sequence models to move away from repetitive, template-like outputs.
Eliminating Bias in Paraphrasing and Semantic Parsing
The benefits of visual anchoring extend far beyond standard text generation into tasks that traditionally rely on text-to-text manipulation.
Unbiased Paraphrasing
Traditional sentence-to-sentence rewriting is heavily constrained by the original author's word choices. To circumvent this evaluation crisis, Chen et al. (2011) used short video clips as a neutral semantic anchor. Multiple observers described the same video independently, generating a highly parallel dataset. Because participants described a scene from scratch rather than modifying an existing sentence, the data captured authentic linguistic variations completely free of direct textual bias.
Lexically Diverse Dialogue
In semantic parsing and dialogue systems, traditional crowdsourcing prompts often use text-based entity labels. To combat the resulting lexical anchoring, Ravichander et al. (2017) replaced text labels with visual icons in their crowdsourcing pipeline. By removing the textual cues, the authors successfully encouraged participants to use a broader range of synonyms and indirect phrasing (e.g., recognizing an icon of a lightbulb and saying "make it brighter" instead of just repeating the word "light").
Preserving Authenticity in Specialized and Multilingual Domains
Visual elicitation is particularly crucial when capturing complex structures, professional jargon, or fluid multilingual speech, where text prompts would otherwise corrupt the natural distribution of language.
- Complex Logical Structures: When benchmarking regular expression synthesis, Chen et al. (2020) utilized abstract diagrams instead of text descriptions to prompt annotators. This forced them to organically formulate descriptions of complex logic, resulting in higher lexical variety and more complex sentence structures that better reflect real-world programming challenges.
- Authentic Professional Jargon: When collecting spoken medical drug prescriptions for spoken language understanding, Kocabiyikoglu et al. (2022) used visual diagrams and pictographs. This prevented the researchers' own text prompts from priming the speakers, ensuring that clinicians and laypeople used their authentic, real-world professional jargon and natural phrasing.
- Multilingual Fluidity: For code-mixed languages (such as Hinglish or Tenglish), text-based instructions often disrupt natural speech patterns. Chandu et al. (2018) used images as anchor points to eliminate lexical entrainment, successfully capturing the fluid syntax and natural vocabulary shifts typical of multilingual communities.
- Defeating "Translationese": In low-resource language data collection, translating from a dominant language like English often introduces "translationese"—stiff, unnatural grammatical structures that mirror the source language. To solve this, Kuwanto et al. (2024) introduced a storyboard-based approach. By prompting native speakers with visual scenes rather than source text, the authors allowed speakers to utilize authentic native syntactic patterns, cultural nuances, and richer vocabulary distributions.
Conclusion
The core insight of visual elicitation is that the format of the prompt dictates the diversity of the response. Textual prompts act as a cognitive gravity well, pulling human annotators toward repetitive and rigid language patterns. By replacing text with visual stimuli, researchers successfully shift the human task from textual manipulation to conceptual description, unlocking the true variety, fluidity, and authenticity of human language for AI training datasets.
Key Foundational Work
Crowd-sourcing NLG Data: Pictures Elicit Better Data (2016) — contributed 2 concept elements to this direction.
The aNALoGuE Challenge: Non Aligned Language GEneration (2016) — contributed 2 concept elements to this direction.
Collecting Highly Parallel Data for Paraphrase Evaluation (2011) — contributed 2 concept elements to this direction.