Visual Question Rewriting for Increasing Response Rate

Jiayi Wei, Xilian Li, Yi Zhang, Xin Eric WangSIGIR 2021 · DOI · OpenReview

Abstract: When a human asks questions online, or when a conversational virtual agent asks human questions, questions triggering emotions or with details might more likely to get responses or answers. we explore how to automatically rewrite natural language questions to improve the response rate from people. In particular, a new task of Visual Question Rewriting(VQR) task is introduced to explore how visual information can be used to improve the new questions. A data set containing around 4K bland questions, attractive questions and images triples is collected. We developed some baseline sequence to sequence models and more advanced transformer based models, which take a bland question and a related image as input and output a rewritten question that is expected to be more attractive. Offline experiments and mechanical Turk based evaluations show that it is possible to rewrite bland questions in a more detailed and attractive way to increase the response rate, and images can be helpful.

Introduction

In online Q&A communities such as Houzz, Reddit, or real estate forums, many user queries go unanswered. Research into these interactions reveals a consistent pattern: "bland" questions (e.g., "What are the dimensions?") are frequently ignored, whereas detailed, specific, or emotionally engaged questions (e.g., "I love this kitchen! What are the dimensions?") are significantly more likely to elicit a response.

However, most users lack the time or copywriting skill to craft these "attractive" queries. To bridge this gap, this paper introduces the task of Visual Question Rewriting (VQR). The goal is to build an AI system that takes a simple user query and a related image as input, then automatically rewrites the query to be more detailed and polite, thereby increasing the likelihood of a response from human experts.

Methodology

The researchers evaluated several architectures to address the VQR task, ranging from traditional recurrent networks to modern pre-trained Transformers.

Baseline Models

  1. RNN Baseline: A standard Sequence-to-Sequence (Seq2Seq) model utilizing Gated Recurrent Units (GRUs) with an attention mechanism.
  2. Transformer Baseline: Based on the UniLM (Unified Language Model) framework, this model adapts BERT-style pre-training for text generation. It employs a specific attention mask that allows the model to encode the input bidirectionally while generating the output unidirectionally.
The Transformer-based model uses a specific attention mask to support bidirectional encoding of the prompt and unidirectional generation of the response.

Visual Integration

To incorporate visual context, the authors extracted image features using a pre-trained ResNet-50. While the RNN model used spatial attention over these features, the Transformer model utilized a more sophisticated approach: Conditional Layer Normalization.

Instead of simply concatenating visual data to the text embeddings, the image features are used to modulate the gain ($\gamma$) and bias ($\beta$) parameters of the Transformer’s normalization layers. This mechanism allows the visual information to "steer" the language generation process, ensuring that the added details are grounded in the specific image context (e.g., mentioning the "wood grain" or "stain color" of a floor).

Conditional Layer Normalization modulates the Transformer's internal states using visual features (C) to guide the generation of image-specific details.

The VQR Dataset

The authors introduced a new dataset specifically for this task, crawled from Houzz.com. The dataset consists of approximately 4,000 triplets containing an image, a "bland" question, and an "attractive" question. Attractive questions were identified as multi-sentence queries that successfully received answers from designers on the platform.

Key Results

Experimental results demonstrated that the Transformer-based models, particularly those leveraging visual context, significantly outperformed the RNN baselines:

  • Quantitative Performance: The Transformer+vis model achieved a BLEU score of 0.547 and a ROUGE-L of 0.885, a massive improvement over the RNN baseline's BLEU of 0.194.
  • Human Evaluation: In a study using Amazon Mechanical Turk, human workers preferred the rewritten questions over the original bland versions 85.8% of the time.
  • The Visual Advantage: When comparing the text-only Transformer to the image-aware version, humans preferred the version with visual input 53.4% of the time, confirming that the model successfully used the photo to add meaningful, attractive details.

Conclusion and Future Work

Visual Question Rewriting represents a novel way to improve human-computer and human-human interaction in online communities. By using conditional normalization to integrate visual cues into pre-trained language models, the VQR system can generate engaging, detailed questions that increase response rates. Future research will focus on expanding the dataset size and exploring how well these models generalize to other domains like fashion or electronics e-commerce.

Summary updated

Abstract

When a human asks questions online, or when a conversational virtual agent asks a human questions, questions triggering emotions or with details might more likely to get responses or answers. we explore how to automatically rewrite natural language questions to improve the response rate form people. In particular, a new task of Visual Question Rewriting (VQR) task is introduced to explore how visual information can be used to improve the new question(s). A data set containing -4K bland&attractive question-images triples is collected. We developed some baseline sequence to sequence models and more advanced transformer-based models, which take a bland question and a related image as input, and output a rewritten question that's expected to be more attractive. Offline experiments and mechanical Turk based evaluations show that it's possible to rewrite bland questions in a more detailed and attractive way to increase response rate, and images can be helpful.
BibTeX
@inproceedings{wei2021visual,
  author = {Jiayi Wei and Xilian Li and Yi Zhang and Xin Eric Wang},
  title = {{Visual Question Rewriting for Increasing Response Rate}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2021},
  doi = {10.1145/3404835.3463114},
}
Figure