Abstract
Traditional recommendation systems focus on maximizing user engagement, often overlooking critical behavioral or ethical constraints. This paper introduces Behavior Constrained Contextual Bandits (BCCB), a framework that allows agents to learn these constraints from observation. By employing a tunable parameter, $\sigma_{online}$, the system balances reward maximization with adherence to learned policies. Evaluated on the MovieLens 20m dataset, BCCB demonstrates the ability to provide high-quality recommendations while strictly following safety guidelines.
Introduction
Multi-Armed Bandits (MAB) are a staple of modern recommendation engines, typically optimized to maximize a single reward metric like click-through rates or user ratings. However, in sensitive domains such as healthcare or child-oriented media, unconstrained optimization can lead to inappropriate or harmful actions. For instance, a system might recommend R-rated content to a minor simply because it predicts a high rating.
While hard-coding rules is a common solution, explicit constraints are not always available or easily formalized. This research addresses the challenge of learning exogenous constraints from expert behavior and navigating the inherent trade-off between following those rules and seeking high rewards in an online environment.
Methodology: Learning from Observation
The BCCB framework extends the Contextual Multi-Armed Bandit (CMAB) model, specifically utilizing Contextual Thompson Sampling (CTS). The process is divided into two phases:
- Teaching Phase: The agent observes a "teacher" (e.g., a domain expert or parent) making decisions. It uses CTS to learn a constraint policy ($\mu^e$) that maps user features (context) to appropriate actions.
- Recommendation Phase: During live deployment, the agent evaluates potential actions using two distinct models: a Reward Model ($\mu_k$) to predict user preference and a Constraint Model ($\mu^e_k$) to ensure alignment with learned behavioral norms.
Balancing Constraints and Rewards
A key innovation of BCCB is the introduction of the $\sigma_{online}$ parameter, which provides a manual "knob" for system designers to control the trade-off between utility and safety.
As shown in the system interface, when $\sigma_{online}$ is set to 0.0, the agent prioritizes the learned constraints above all else. Conversely, a value of 1.0 would cause the agent to ignore constraints in favor of maximizing rewards. This flexibility allows for "intrinsic explainability," as the model's linear weights can highlight which features (e.g., a "Horror" genre tag for a young user) triggered a constraint violation.
Evaluation and Key Results
The researchers tested BCCB using the MovieLens 20m dataset, focusing on user demographics and movie genres. To evaluate the learning process, they established a synthetic ground truth for constraints based on age-appropriate genre frequencies.
The results indicate that BCCB is highly effective:
- Rapid Learning: The agent developed high-quality policies using a teaching set of only 200 movies.
- Competitive Performance: BCCB achieved reward levels comparable to systems using hard-coded constraints, proving that learning from examples is a robust alternative to manual programming.
- High Adherence: The system consistently made decisions that aligned with learned policies, even while exploring new content to optimize user ratings.
Conclusion
BCCB represents a significant step toward safer AI by treating "appropriateness" as a learnable objective. While the current study relies on synthetic constraints and linear assumptions, it lays the groundwork for more complex applications. Future research will explore non-linear models and dynamic constraint updates to better reflect evolving societal norms and legal requirements.
Summary updated