Preference-Guided Multi-Objective UI Adaptation

Yao Song, Christoph Gebhardt, Yi-Chi Liao, Christian Holz

Abstract

Designing effective user interface (UI) layouts in Mixed Reality (MR) environments presents a complex optimization challenge. Designers must balance conflicting objectives such as ergonomic comfort, reachability, and semantic organization. Traditional Multi-Objective Optimization (MOO) methods often overwhelm users with a "Pareto frontier" of potential designs, requiring manual selection or complex weighting. This paper presents a human-in-the-loop optimization framework that infers user preferences from minimal manual adjustments and utilizes a tiered priority algorithm to generate personalized, optimal layouts across the entire interface.

Introduction

In 3D Mixed Reality, the availability of nearly unlimited spatial real estate for virtual windows introduces a significant usability hurdle: the "manual organization tax." While users can place apps anywhere, manually arranging them to minimize neck strain while maintaining organizational logic is tedious. Automated systems using Multi-Objective Optimization (MOO) attempt to solve this by satisfying goals like minimizing physical fatigue and grouping related applications. However, MOO typically produces a set of trade-offs rather than a single solution. Navigating this Pareto frontier is difficult for non-experts, who often struggle to translate their internal preferences into numerical weights.

Methodology

The proposed system addresses the preference-elicitation gap through a three-step integrated workflow designed for real-time MR interaction.

1. Implicit Preference Inference

Rather than asking users to assign weights to design goals, the system observes user behavior. When a user moves a single UI element (e.g., a music player), the system analyzes how that movement affects five key design objectives. For instance, moving a window closer to a physical phone suggests a high preference for "Semantic Agreement." To filter out accidental movements or "noise," the system employs a Triangular Moving Average to aggregate and smooth these inferred preferences over time.

2. Priority-Based Optimization (PL-NSGA-II)

To handle the trade-offs between objectives, the system categorizes goals into High, Mid, and Low priority tiers. It utilizes a Mixed Pareto-Lexicographic MOO (PL-NSGA-II) algorithm. This approach ensures that high-priority goals are satisfied first. By focusing the computational search budget on the most critical tiers, the algorithm avoids wasting resources on designs that satisfy low-priority goals at the expense of essential ones.

3. Automatic Selection and Semantic Mapping

The system identifies the layout on the generated Pareto frontier that is mathematically closest to the user’s manual adjustments in the objective space. To enhance intelligent placement, the framework incorporates WordNet to calculate semantic associations. This allows the system to understand that a video player is semantically related to a tablet, enabling it to make informed placement decisions for windows the user has not yet interacted with.

Evaluation

The system was evaluated using high-fidelity 3D scans of a coffee shop and a home office from the SemanticAdapt codebase. A user study involving 12 participants compared the preference-guided approach against manual adjustment and standard MOO selection methods in both "free personalization" and "targeted" tasks.

Key Results

  • Efficiency: Participants moved significantly fewer UI elements to reach a satisfactory layout compared to baseline methods.
  • User Satisfaction: In head-to-head rankings, users consistently preferred layouts generated by the preference-guided method over standard Pareto selection.
  • Algorithmic Convergence: Quantitative analysis confirmed that the PL-NSGA-II algorithm produced solution candidates significantly closer to the user’s implicit reference points, particularly for high-priority objectives.

Discussion and Future Work

The study demonstrates that "actions speak louder than weights" in UI personalization. By observing interaction, the system captures intent more accurately than traditional sliders. However, several areas for improvement remain:

  • Constraint Handling: Currently, the optimizer may "correct" a window a user just moved if it improves the global score. Future iterations could treat manual moves as "hard constraints."
  • Uncertainty Modeling: The current deterministic model assumes every move reflects a clear preference. Integrating Bayesian modeling could better account for exploratory behavior or user error.
  • Scalability: While effective for 7 widgets and 5 objectives, further research is needed to ensure real-time performance in professional MR setups involving dozens of windows.

Conclusion

Preference-guided multi-objective adaptation offers a promising path toward frictionless MR environments. By combining implicit preference inference with lexicographic optimization, the system reduces the cognitive and physical load of UI management, allowing users to focus on their primary tasks rather than interface maintenance.

Summary updated

BibTeX
@inproceedings{song2025preference,
  author = {Yao Song and Christoph Gebhardt and Yi-Chi Liao and Christian Holz},
  title = {{Preference-Guided Multi-Objective UI Adaptation}},
  booktitle = {UIST},
  year = {2025},
  doi = {10.1145/3746059.3747645},
}
Figure