TL;DR
This paper tackles the "cold start" problem in adaptive political surveys, where a lack of initial user data hinders question selection. They demonstrate that GPT-4 can generate realistic synthetic user interaction data, which, when used to pre-train the survey model, significantly improves early user experience and accuracy. This offers a promising, low-cost solution for building effective adaptive surveys in data-scarce domains.
What Problem Does This Paper Address?
Adaptive questionnaires dynamically select the next question based on a user's previous answers, aiming to gather information efficiently. This is particularly useful in political science for tools like Voting Advice Applications (VAAs) or wiki surveys, where users might not complete long questionnaires. However, a major hurdle is the "cold start" problem: these adaptive systems rely on a statistical model trained on user interaction data, which is often unavailable at the beginning of a survey. This lack of initial data leads to poor question selection and a suboptimal user experience for early participants. The authors aim to solve this by using Large Language Models (LLMs) to generate synthetic training data.
Methods
The core of the paper involves two main experiments:
- Synthetic Data Generation: They use GPT-4, an off-the-shelf LLM, to simulate political candidates answering a survey. This is achieved through careful "prompt engineering," where GPT-4 is instructed to adopt the persona of a specific Swiss political party and provide answers on a numerical scale. They explore variations of this synthetic data, including averaging responses per party (GPTmeans) and creating interpolated "voter" profiles (GPTvoters) to better mimic real-world distributions.
- Adaptive Questionnaire Simulation: They simulate users interacting with an adaptive questionnaire. The key innovation here is comparing the performance of a questionnaire model initialized randomly (the "cold start" baseline) against models pre-trained with different versions of the GPT-4 generated synthetic data. They evaluate performance on two downstream tasks: missing value imputation (how well the model can predict answers for users who drop out) and candidate recommendation (how accurately it can identify suitable candidates in a VAA).
Datasets
- Smartvote Dataset: The primary dataset used is from the Swiss Voting Advice Application (VAA) "Smartvote" for the 2023 Swiss National Elections. This dataset contains responses from 1,029 candidates and 25,783 voters to 75 political questions. The authors focus on data from the canton of Zurich.
- Synthetic Data (GPT, GPTmeans, GPTvoters): Generated using GPT-4 based on the Smartvote questionnaire and party profiles.
- Benchmark Dataset (Candidates): The real candidate data from Smartvote is used as a benchmark for evaluating the quality of synthetic data and the performance of a fully trained model.
Key Insights and Ideas
- LLMs as Synthetic Data Generators: GPT-4 can convincingly emulate the responses of political actors from different parties, providing a viable source of training data where real data is scarce.
- Mitigating the Cold Start: Pre-training adaptive questionnaire models with LLM-generated data significantly improves performance for early users, effectively reducing the cold start problem.
- Data Nuance vs. Consistency: While GPT-4 generates data that is generally closer to party means than real candidates, it tends to be less extreme and more consistent, potentially lacking the full spectrum of real-world political diversity.
- Break-Even Points: The initial advantage of pre-trained models diminishes over time as real user interactions accumulate. The point at which a randomly initialized model catches up (the "break-even point") varies depending on the downstream task and the number of questions answered per user.
Key Results
- Data Quality: GPT-4 generated synthetic data points were within one standard deviation of real candidates' answers for 85.3% of questions. However, the synthetic data showed less variance and a tendency to be closer to party means than real candidates.
- Improved Early Performance: Models pre-trained with GPT-4 data showed significantly better performance for early users compared to a randomly initialized model.
- For missing value imputation (RMSE), pre-trained models started with an RMSE of ~0.315-0.327, compared to ~0.420 for the cold start model.
- For candidate recommendation (CRA), pre-trained models achieved an initial CRA of ~42.3-43.2%, a substantial improvement over the cold start model's ~24.8%.
- Break-Even Points:
- For missing value imputation, the break-even point (where cold start matches pre-trained) occurred after approximately 85-895 users, with fewer users needed when users answered more questions.
- For candidate recommendation, the relationship was more complex, with break-even points occurring after 290-650 users, depending on the number of questions answered.
- Extremity Bias: Pre-training with synthetic data introduced a bias towards recommending more moderate candidates, especially when fewer questions were answered.
Limitations and Future Work
- Domain Specificity: The method relies on LLMs having domain knowledge. GPT-4's success in Swiss politics might not generalize to all political systems or other domains (e.g., movies, products) where LLMs might struggle with item consumption.
- Bias Introduction: While synthetic data is eventually replaced by real data, there's a risk of path dependencies where the question selection policy might reinforce initial biases. The authors found no increased bias compared to the cold start setting in their analysis, but acknowledge more complex biases could exist.
- Simulation Setup: The simulation focused on a specific uncertainty-based question selection strategy and didn't include aspects like exploitation/exploration trade-offs common in recommender systems.
- Parameter Tuning: The simulation introduced a "replacement parameter" (γ) for how quickly synthetic data is removed. Finding the optimal value for γ requires further systematic investigation.
- Future Work: The authors suggest extending their interpolation method for synthetic data, validating it with other datasets, and exploring more sophisticated question selection strategies. They also highlight the need to better predict break-even points for practical deployment.
Summary updated