TL;DR
This paper empirically investigates the role of images in Stack Overflow questions. They found that images, particularly of user interfaces showing undesired outputs, are crucial for understanding 68% of questions and significantly increase the likelihood of receiving accepted answers, though they don't speed up response times.
What Problem Does This Paper Address?
Software developers increasingly use platforms like Stack Overflow to ask and answer technical questions. While text is the primary mode of communication, images are frequently included. Prior research suggests these images are meaningful and offer complementary information, but how they are used and their actual impact on question quality and answer reception remained largely unknown. This paper aims to fill that gap by empirically studying the characteristics of images, their usage across different question types, and their influence on receiving answers on Stack Overflow.
Methods
The researchers conducted a multi-faceted empirical study:
- Qualitative Analysis (RQ1): They manually classified a representative sample of 384 questions with images to understand image characteristics. This involved categorizing image sources (e.g., desktop application, web browser), content (e.g., user interface, diagram, error code), and purpose (e.g., undesired output, additional context). They also assessed the "essentialness" of images for question comprehension.
- Quantitative Analysis (RQ2 & RQ3):
- Question Types: They classified questions into nine types (e.g., how-to, discrepancy, error) based on existing schemas and analyzed the distribution of images across these types. They also compared description length and questioner experience between questions with and without images.
- Answer Reception: They analyzed three metrics: whether a question received any answer, whether it received an accepted answer, and the time taken to receive the first answer. Statistical tests (Chi-square, Bonferroni's multiple comparison) were used to validate hypotheses.
Datasets
The study utilized a large dataset of Stack Overflow questions mined from the SOTorrent dataset as of December 2019. This dataset comprised approximately 18.7 million questions. For the qualitative analysis, a statistically representative sample of 384 questions containing images was drawn. A control group of 384 questions without images was also created for comparison.
Key Insights and Ideas
- Images are Essential, Not Just Decorative: A significant majority (68%) of images were found to be essential for understanding the question, meaning the question couldn't be comprehended without them. This highlights images as a critical component of technical communication.
- Undesired Outputs are a Major Driver: The most common purpose for sharing images was to show an "undesired output" (45%), suggesting developers use visuals to clearly illustrate problems they are facing.
- User Interfaces Dominate: User interfaces were the most frequent image content (58%), reinforcing the idea that visual representations of software are key to debugging and problem-solving.
- Discrepancy Questions Benefit Most: While "how-to" questions are the most common overall, "discrepancy" questions (where unexpected behavior needs explanation) are disproportionately more likely to include images.
- Experience Matters: Questions with images were more likely to be submitted by experienced users, suggesting a strategic use of visuals by those with a deeper understanding of effective communication.
Key Results
- Image Characteristics: Desktop applications (47%) and web browsers (26%) were the most common image sources. User interfaces (58%) were the most common content, and undesired output (45%) was the most common purpose.
- Question Types: "How-to" (46%) and "discrepancy" (24%) were the most frequent question types containing images. Discrepancy questions were 6% more likely to contain images than questions without them.
- Description Length: No significant difference was found in the description length between questions with and without images.
- Answer Reception:
- Questions with images were significantly more likely to receive accepted answers (56% vs. 47% overall).
- The presence of images did not significantly speed up the time to receive the first answer.
- The presence of images did not significantly increase the probability of receiving any answer.
Limitations and Future Work
- External Validity: The study is limited to Stack Overflow, so findings might not generalize to other Q&A forums.
- Image Retrieval: The study relied on image file extensions, which might miss some image types.
- First Image Focus: Only the first image in a post was analyzed, assuming it represents the primary visual information.
- Subjectivity in Coding: Manual classification of image characteristics and question types introduced potential subjectivity, though rigorous inter-rater agreement measures were employed.
- Causality: While correlations are shown (e.g., images and accepted answers), direct causality is hard to establish without controlling for confounding factors.
Future work suggested includes:
- Investigating the causality between images and answers more deeply.
- Conducting developer surveys to understand how images might increase cognitive load.
- Studying the role of images in answers themselves.
- Exploring the effect of images on downstream tasks like automatic question generation and question relatedness, leveraging advancements in OCR and computer vision.
Summary updated