Evaluating LLMs Towards Automated Assessment of Privacy Policy Understandability

Keika Mori, Daiki Ito, Takuro Fukunaga, Takuya Watanabe, Yuta Takata, Masaki Kamizono, Tatsuya Mori2025 · DOI

Abstract

Companies publish privacy policies to improve transparency regarding the handling of personal information.A discrepancy between the description of the privacy policy and the user's understanding can lead to a risk of a decrease in trust.Therefore, in creating a privacy policy, the user's understanding of the privacy policy should be evaluated.However, the periodic evaluation of privacy policies through user studies takes time and incurs financial costs.In this study, we investigated the understandability of privacy policies by large language models (LLMs) and the gaps between their understanding and that of users, as a first step towards replacing user studies with evaluation using LLMs.Obfuscated privacy policies were prepared along with questions to measure the comprehension of LLMs and users.In comparing the comprehension levels of LLMs and users, the average correct answer rates were 85.2% and 63.0%, respectively.The questions that LLMs answered incorrectly were also answered incorrectly by users, indicating that LLMs can detect descriptions that users tend to misunderstand.By contrast, LLMs understood the technical terms used in privacy policies, whereas users did not.The identified gaps in comprehension between LLMs and users, provide insights into the potential of automating privacy policy evaluations using LLMs.TABLE II QUESTIONS AND ANSWER OPTIONS TO CHECK COMPREHENSION LEVEL. # Question TextAnswer Options Q1 Select all options that are correct for the information A. Use email addresses for account management.items to be collected and used by PRIVACY Co., Ltd.B. Use location data for advertisement.along with their purposes.If choosing option E, C. Use device data for step count estimation.answer only option E without the others.D. Use email addresses for advertisement.E. Cannot be determined from the privacy policy (no description).Q2 Select all options that are correct regarding the purpose A. To measure advertising effectiveness.of the personal data provided by PRIVACY Co., Ltd.B. To develop new services.to other companies.For option D, answer only option C. To contact with customers.D without the others.D. Cannot be determined from the privacy policy (no description).Q3 Select all options that are correct for security measures A. Measures to prevent loss of devices that handle personal data.taken by PRIVACY Co., Ltd.For option D, answer B. Maintaining security certification by external organization.only option D
BibTeX
@misc{mori2025evaluating,
  author = {Keika Mori and Daiki Ito and Takuro Fukunaga and Takuya Watanabe and Yuta Takata and Masaki Kamizono and Tatsuya Mori},
  title = {{Evaluating LLMs Towards Automated Assessment of Privacy Policy Understandability}},
  year = {2025},
  doi = {10.14722/usec.2025.23009},
}

Loading related content...

Figure