Abstract: When a human asks questions online, or when a conversational virtual agent asks human questions, questions triggering emotions or with details might more likely to get responses or answers. we explore how to automatically rewrite natural language questions to improve the response rate from people. In particular, a new task of Visual Question Rewriting(VQR) task is introduced to explore how visual information can be used to improve the new questions. A data set containing around 4K bland questions, attractive questions and images triples is collected. We developed some baseline sequence to sequence models and more advanced transformer based models, which take a bland question and a related image as input and output a rewritten question that is expected to be more attractive. Offline experiments and mechanical Turk based evaluations show that it is possible to rewrite bland questions in a more detailed and attractive way to increase the response rate, and images can be helpful.
Introduction
In online Q&A communities such as Houzz, Reddit, or real estate forums, many user queries go unanswered. Research into these interactions reveals a consistent pattern: "bland" questions (e.g., "What are the dimensions?") are frequently ignored, whereas detailed, specific, or emotionally engaged questions (e.g., "I love this kitchen! What are the dimensions?") are significantly more likely to elicit a response.
However, most users lack the time or copywriting skill to craft these "attractive" queries. To bridge this gap, this paper introduces the task of Visual Question Rewriting (VQR). The goal is to build an AI system that takes a simple user query and a related image as input, then automatically rewrites the query to be more detailed and polite, thereby increasing the likelihood of a response from human experts.
Methodology
The researchers evaluated several architectures to address the VQR task, ranging from traditional recurrent networks to modern pre-trained Transformers.
Baseline Models
- RNN Baseline: A standard Sequence-to-Sequence (Seq2Seq) model utilizing Gated Recurrent Units (GRUs) with an attention mechanism.
- Transformer Baseline: Based on the UniLM (Unified Language Model) framework, this model adapts BERT-style pre-training for text generation. It employs a specific attention mask that allows the model to encode the input bidirectionally while generating the output unidirectionally.
Visual Integration
To incorporate visual context, the authors extracted image features using a pre-trained ResNet-50. While the RNN model used spatial attention over these features, the Transformer model utilized a more sophisticated approach: Conditional Layer Normalization.
Instead of simply concatenating visual data to the text embeddings, the image features are used to modulate the gain ($\gamma$) and bias ($\beta$) parameters of the Transformer’s normalization layers. This mechanism allows the visual information to "steer" the language generation process, ensuring that the added details are grounded in the specific image context (e.g., mentioning the "wood grain" or "stain color" of a floor).
The VQR Dataset
The authors introduced a new dataset specifically for this task, crawled from Houzz.com. The dataset consists of approximately 4,000 triplets containing an image, a "bland" question, and an "attractive" question. Attractive questions were identified as multi-sentence queries that successfully received answers from designers on the platform.
Key Results
Experimental results demonstrated that the Transformer-based models, particularly those leveraging visual context, significantly outperformed the RNN baselines:
- Quantitative Performance: The Transformer+vis model achieved a BLEU score of 0.547 and a ROUGE-L of 0.885, a massive improvement over the RNN baseline's BLEU of 0.194.
- Human Evaluation: In a study using Amazon Mechanical Turk, human workers preferred the rewritten questions over the original bland versions 85.8% of the time.
- The Visual Advantage: When comparing the text-only Transformer to the image-aware version, humans preferred the version with visual input 53.4% of the time, confirming that the model successfully used the photo to add meaningful, attractive details.
Conclusion and Future Work
Visual Question Rewriting represents a novel way to improve human-computer and human-human interaction in online communities. By using conditional normalization to integrate visual cues into pre-trained language models, the VQR system can generate engaging, detailed questions that increase response rates. Future research will focus on expanding the dataset size and exploring how well these models generalize to other domains like fashion or electronics e-commerce.
Summary updated