Refining App Reviews: Dataset, Methodology, and Evaluation

Amrita Singh, Chirag Jain, Mohit Chaudhary, Preethu Rose AnishEMNLP 2024

TL;DR

This paper introduces RARE, a novel benchmark dataset and methodology for refining mobile app reviews. By cleaning up noisy, error-filled user feedback, RARE aims to make app reviews more actionable for developers, leading to faster improvements and better user experiences.

What Problem Does This Paper Address?

Mobile app stores are flooded with user reviews, which are a goldmine of information for developers. However, these reviews are often riddled with typos, grammatical errors, informal language, and complex sentences, making them difficult and time-consuming to analyze. This hinders developers from efficiently identifying bugs, understanding user sentiment, and prioritizing feature enhancements, ultimately slowing down the app improvement cycle. The paper addresses this by proposing a systematic way to refine these reviews into a more structured and understandable format.

Methods

The core of the paper's methodology involves a collaborative approach to review refinement, combining the power of Large Language Models (LLMs) with human expertise.

  1. Operations Identification: The authors first manually analyzed a sample of reviews to identify five common issues: grammatical errors, complex phrasing, irrelevant information, word/information rearrangement, and sentence structure issues.
  2. Prompt Engineering: Based on these identified operations, they designed six distinct prompts to guide GPT-3.5-Turbo in refining reviews.
  3. Pilot Refinement: GPT-3.5-Turbo was used with these prompts to generate initial refined reviews.
  4. Quantitative and Qualitative Analysis: These refined reviews were then evaluated using both automatic metrics (like FKGL, FKRE, SARI) and human evaluation by experienced software engineers. This helped identify the most effective prompt.
  5. Human Correction: The best-performing LLM-generated reviews were further refined and corrected by human experts to create a high-quality "gold standard" corpus.
  6. LLM Fine-tuning and Silver Corpus Generation: The gold standard corpus was used to fine-tune various state-of-the-art LLMs. The best-performing model (Flan-T5) was then used to generate a larger "silver standard" corpus.

Datasets

The paper introduces the RARE (Repository for App review REfinement) dataset, which consists of:

  • Gold Corpus: 10,000 pairs of original and human-refined app reviews, collected from 10 different mobile applications on the Google Play Store.
  • Silver Corpus: 10,000 pairs of original and LLM-refined app reviews, generated by the best-performing model (Flan-T5) from the Apple App Store.

This dataset is designed to benchmark LLMs and other machine learning models for app review refinement.

Key Insights and Ideas

  • LLMs + Humans = Best Results: The most effective review refinement comes from a collaborative process, where LLMs handle the bulk of the work, and human experts provide crucial corrections and validation.
  • Prompt Engineering Matters: The way prompts are phrased significantly impacts the quality of LLM-generated refinements. Simple, clear prompts were found to be more effective than overly complex ones.
  • Refinement Improves Readability: Refined reviews are demonstrably shorter, clearer, and easier to understand, as evidenced by improved readability scores (FKGL, FKRE) and reduced word counts.
  • Domain-Specific Knowledge is Key: While LLMs are powerful, they sometimes struggle with the nuances of app reviews, leading to issues like deleting relevant information or oversimplifying. Human oversight is essential to bridge this gap.
  • Refinement Aids Downstream Tasks: Preliminary experiments show that refined reviews lead to better performance in tasks like classifying reviews into bug reports or feature requests.

Key Results

  • Flan-T5 as Top Performer: Flan-T5 emerged as the best-performing LLM for app review refinement, excelling in both automatic and human evaluations across metrics like meaning preservation (BP), simplification (Qd), and overall quality (Qa).
  • Significant Readability Improvement: The FKGL (Flesch-Kincaid Grade Level) scores for refined reviews were consistently lower than raw reviews across all apps, indicating improved readability. For example, the gold corpus showed a significant decrease in FKGL scores after refinement.
  • Conciseness Achieved: Refined reviews were typically 20-25% shorter than original reviews, demonstrating increased conciseness.
  • Improved Classification Accuracy: A multi-label classification task showed an improvement in weighted average F1 score from 0.81 for raw reviews to 0.89 for refined reviews.

Limitations and Future Work

  • Dataset Size: While RARE is a substantial dataset, the authors acknowledge that a larger dataset could further enhance model robustness.
  • LLM Limitations: GPT-3.5-Turbo, despite its capabilities, sometimes deleted relevant information or oversimplified reviews, highlighting the need for human correction.
  • Future Work: The authors suggest exploring the extraction of non-functional requirements from app reviews and assessing how refinement impacts this process. They also propose summarizing extracted requirements as a future direction.

Summary updated

BibTeX
@inproceedings{singh2024refining,
  author = {Amrita Singh and Chirag Jain and Mohit Chaudhary and Preethu Rose Anish},
  title = {{Refining App Reviews: Dataset, Methodology, and Evaluation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
}
Figure