TL;DR
This paper investigates whether spoken and written image descriptions differ systematically, even when elicited by similar tasks. The authors found that spoken descriptions tend to be longer, use more adverbs and pseudo-quantifiers, and reflect speaker certainty more than written ones. This has implications for how we design and evaluate image description systems, particularly for accessibility applications.
What Problem Does This Paper Address?
Automatic image description systems are a rapidly developing area of AI, with applications ranging from content indexing to assisting visually impaired users. These systems are typically trained and evaluated on large datasets of written image descriptions. However, many real-world applications, such as those using screen readers, deliver descriptions verbally. Linguistic research has long established that spoken and written language often diverge in systematic ways, influenced by factors like spontaneity, audience, and processing constraints. This paper asks: do these linguistic differences manifest in image descriptions, and if so, how? Understanding these differences is crucial for developing image description systems that produce natural-sounding and effective outputs, especially for spoken modalities.
Methods
The researchers conducted an exploratory study comparing spoken and written image descriptions in both English and Dutch. They utilized existing datasets and collected new data for the Dutch spoken descriptions. Their approach involved:
- Tokenization, Tagging, and Parsing: Using SpaCy, they processed the descriptions to analyze linguistic features.
- Metric Computation: They computed a range of linguistic metrics, drawing from prior research on spoken vs. written language, including:
- Token Length: Average number of syllables per word.
- Description Length: Average number of tokens and syllables per description.
- Mean-Segmental Type-Token Ratio (MSTTR): A measure of lexical diversity.
- Attributive Adjectives: Adjectives directly modifying nouns.
- Adverbs: Words modifying verbs, adjectives, or other adverbs.
- Prepositions: Words indicating spatial or temporal relationships.
- Consciousness-of-projection terms: Words indicating the observer's perspective (e.g., "apparently," "seems").
- Self-reference terms: First-person pronouns (e.g., "I," "me," "my").
- Positive allness terms: Words like "all," "each," "every."
- Negations: Words like "no," "not," "never."
- Pseudo-quantifiers: Words indicating approximate quantity (e.g., "many," "few," "lots").
- Propositional Idea Density (PID): A measure of how much propositional information is conveyed per word.
Datasets
The study used the following datasets:
- Written English:
- Flickr30K: Contains images with 5 written descriptions per image.
- MS COCO: Similar to Flickr30K, with 5 written descriptions per image, but with specific instructions to avoid "There is..." constructions.
- Spoken English:
- Places Audio Caption Corpus: A large dataset of approximately 230,000 spoken descriptions for images, automatically transcribed with an estimated 20% word error rate.
- Written Dutch:
- Data collected by van Miltenburg et al. (2017), consisting of Dutch descriptions for Flickr30K validation and test sets.
- Spoken Dutch:
- Newly collected data from a lab experiment where 45 Dutch students described images, with transcriptions and annotations.
The authors note that while the image sets for spoken and written data are not identical, they aimed for comparability and acknowledge this as a limitation.
Key Insights and Ideas
- Modality Matters: The study confirms that the medium of communication (spoken vs. written) influences language use, even in a specific domain like image description.
- Spoken Language is More "Involved": The increased use of consciousness-of-projection and self-reference terms in spoken descriptions suggests a greater tendency for speakers to "show themselves" and frame observations from their own perspective, aligning with theories of "involvement" in communication.
- Spoken Language is More Hedged/Uncertain: The higher frequency of pseudo-quantifiers and allness terms in spoken descriptions might indicate a greater tendency to express uncertainty or soften claims, perhaps due to the spontaneous nature of speech.
- Propositional Density Differences: Spoken English descriptions were found to have lower propositional idea density, meaning they used more words to convey the same amount of factual information compared to written descriptions. This suggests spoken language might prioritize other communicative functions, like pragmatic messaging or engagement, over pure information density.
Key Results
English:
- Description Length: Spoken descriptions were nearly twice as long as written ones.
- Adverbs: Spoken descriptions contained significantly more adverbs (3-4 times more).
- Pseudo-quantifiers & Allness Terms: Spoken descriptions used more of these terms.
- Consciousness-of-projection & Self-reference Terms: Spoken descriptions showed a higher frequency of these terms.
- Propositional Idea Density (PID): Spoken descriptions had a lower PID, indicating less information density.
- Token Length & MSTTR: No clear differences were found in average token length or lexical diversity.
- Attributive Adjectives & Prepositions: Mixed results, with no clear overall difference.
Dutch:
- Description Length: Spoken descriptions were longer, but the difference was less pronounced than in English (about 2 tokens longer on average).
- Adverbs, Pseudo-quantifiers, Allness Terms, Consciousness-of-projection, Self-reference Terms: Similar to English, spoken Dutch descriptions showed higher frequencies of these features.
- Attributive Adjectives & Prepositions: In contrast to English, Dutch spoken descriptions showed fewer attributive adjectives and prepositions per 1000 tokens compared to written ones, aligning more closely with Drieman's original findings.
- Propositional Idea Density (PID): Little difference was found between spoken and written Dutch descriptions, unlike the English results. The authors suggest this might be due to translation issues with the PID rater.
Limitations and Future Work
- Dataset Comparability: The study used different sets of images for spoken and written data, which could influence description length and content.
- Automatic Transcription Errors: The spoken English data had a 20% word error rate from automatic transcription, potentially introducing biases.
- Controlled Replication: The authors emphasize the need for a controlled replication study using parallel spoken and written descriptions for the same images to isolate the effect of modality more definitively.
- User Preference: Future work should investigate whether users prefer spoken or written descriptions and why.
- Broader Language Scope: The study was limited to English and Dutch; exploring other languages is necessary to generalize findings.
- Qualitative Analysis: While the study was quantitative, manual inspection of individual descriptions could reveal interesting phenomena missed by automated metrics.
- Subjectivity and Style: Further exploration into how subjective language and different stylistic choices (e.g., humor, emotion) are affected by modality is warranted.
Summary updated