TL;DR
This paper introduces a novel reinforcement learning (RL) framework for personalized home fitness recommendations. By using a user simulator to generate synthetic data and a carefully designed reward function, the RL agent learns to create effective workout sequences that significantly increase user engagement compared to a baseline approach.
What Problem Does This Paper Address?
Maintaining long-term user engagement with fitness applications is a significant challenge. While many apps exist, they often rely on pre-defined workout plans, offering limited personalization. The COVID-19 pandemic highlighted the need for effective at-home fitness solutions, as gym closures led to increased demand for digital alternatives. However, simply offering a catalog of exercises isn't enough; users need personalized guidance that adapts to their evolving needs and preferences to stay motivated. This paper addresses the gap in personalized, dynamic fitness recommendations that can foster sustained user engagement.
Methods
The core of this work is a Reinforcement Learning (RL) framework designed to recommend sequences of bodyweight exercises. The framework consists of:
- RL Agent: A neural network trained to learn a policy for recommending exercises.
- Gym Environment: This simulates the user and the overall environment. Crucially, it includes a User Simulator that generates synthetic user data and feedback. This simulator is vital for training the RL agent, overcoming the challenge of data scarcity with real users. The simulator is tuned to mimic realistic user behaviors and feedback based on analysis from an initial user trial.
- Action Masking: Used to constrain the agent's choices, for example, by preventing recommendations of exercises that require equipment the user doesn't have or exercises the user has previously disliked.
- Reward Function: A sophisticated, multi-component reward system designed in consultation with sports scientists. It comprises several sub-rewards that capture different aspects of a "good" workout session, including:
- Intra-session Diversity: Encourages variety within a single workout.
- Inter-session Diversity: Promotes variety across different workout sessions.
- Session Matching Fitness Level (FL): Aligns exercise difficulty with the user's assessed fitness level.
- Session Matching User Workout Goal: Ensures exercises align with the user's stated goals (cardio/strength).
- Focus Muscles (FM): Prioritizes exercises that target the user's selected primary and secondary muscles.
- Muscle Fatigue: Penalizes excessive repetition of exercises that fatigue the same muscle groups.
- Wrists: Penalizes exercises that put excessive strain on the wrists.
The RL agent is trained using Proximal Policy Optimization (PPO), a standard and effective RL algorithm.
Datasets
The paper does not explicitly mention a large, pre-existing dataset for training the RL model. Instead, it highlights the creation of a User Simulator to generate synthetic data. This simulator is informed by data gathered during an initial exploratory user trial conducted in 2021, which helped in modeling realistic user feedback and behavior. The framework was evaluated through simulations using this User Simulator and then in a real-world trial.
Key Insights and Ideas
- The Power of Simulation for RL in Niche Domains: The paper demonstrates that a well-designed user simulator can effectively overcome data scarcity issues, enabling the training of complex RL models for domains like personalized fitness recommendations where real-world data is expensive and time-consuming to collect.
- Multi-faceted Reward Engineering is Crucial: A simple reward signal is insufficient. The paper's success hinges on a carefully crafted reward function that balances multiple, sometimes competing, objectives (e.g., diversity vs. fitness level matching). This highlights the importance of domain expertise (sports science) in designing effective RL reward systems.
- Personalization Drives Engagement: The results strongly suggest that personalized recommendations, learned through RL, lead to sustained user satisfaction and engagement over time, contrasting with the degradation seen in baseline methods.
- Sequential Nature of Workouts Matters: Unlike some recommendation tasks, the order and progression of exercises in a workout are critical for user experience and effectiveness. The RL approach, by recommending exercises sequentially, naturally captures this dependency.
Key Results
- Simulation Performance: The RL model significantly outperformed a Particle Swarm Optimization (PSO) baseline in simulated user trials, achieving approximately 20% higher average session quality scores across various metrics.
- Real-World Trial Performance: In a 15-week live trial with 69 users, the RL framework led to significantly higher user satisfaction as measured by the PACES-8 questionnaire. The RL condition showed a 7% improvement in the mean PACES-8 response compared to the baseline.
- Sentiment Analysis: User feedback comments for RL-recommended sessions had a significantly higher positive sentiment score (23.13% higher) and an overall compound score that was 48.88% higher than for the baseline.
- Engagement Over Time: Analysis of weekly average satisfaction showed that while PSO satisfaction tended to decrease over time, RL satisfaction tended to increase, indicating better long-term engagement.
Limitations and Future Work
- Reliance on User Simulator: While effective, the RL model was trained entirely on simulated data. The authors acknowledge that while the simulator was aligned with real-world data, further fine-tuning with real human feedback could improve performance.
- Limited Direct Feedback: The mobile app design limited direct per-exercise feedback, requiring inference from user actions (skipping/changing) and session-level satisfaction.
- Crossover Trial Design: The crossover trial design, while useful for comparison, hindered a direct assessment of long-term engagement.
- Future Work: The authors suggest exploring methods like "RL from human feedback" to further fine-tune the model and replace handcrafted rewards with empirically learned value functions. They also aim to reduce the motivation gap identified in previous work, encouraging longer-term adoption of healthy lifestyles.
Summary updated