Multi-Objective Quality of Experience Optimization in Video Streaming

16 concepts14 papers2 proposals

When you watch a video online, your satisfaction depends on a delicate balance of several moving parts. You want the highest possible resolution, but you also want the video to start instantly, play without freezing (stalling), and avoid sudden, jarring drops in clarity.

For a long time, video delivery algorithms treated these goals in isolation—often focusing purely on maximizing throughput or keeping the playback buffer filled. However, optimizing for a single metric inevitably degrades the others. If an algorithm aggressively pursues peak bitrates, it risks depleting the playback buffer and causing a freeze. Conversely, if it plays it too safe to avoid stalls, the visual quality suffers.

To solve this, modern video streaming has shifted toward Multi-Objective Quality of Experience (QoE) Optimization. Instead of maximizing a single network metric, this approach mathematically unifies competing performance indicators into a single, holistic objective function that mirrors actual human perception.


The Core Streaming Conflict

To understand why multi-objective optimization is necessary, we must look at the fundamental trade-offs inherent to video delivery over volatile networks. As noted by Huan Peng et al., live streaming presents a "trilemma" where algorithms must constantly balance visual quality, playback smoothness, and end-to-end latency.

          [ High Visual Quality ]
                   /   \
                  /     \
                 /       \
 [ Low Latency ] --------- [ Zero Playback Stalls ]

Each corner of this triangle pulls against the others:

  • Quality vs. Stalls: Choosing a higher bitrate chunk improves visual clarity but takes longer to download. If the download time exceeds the playback duration of the current buffer, the video stalls. As Shuai Hao et al. point out, optimizing aggressively for bitrate directly increases the risk of buffer depletion.
  • Latency vs. Stalls: Maintaining a large buffer of downloaded video segments protects against network drops, ensuring continuous playback. However, a larger buffer means the viewer is watching events further in the past. For interactive applications like live sports or social TV, this latency is unacceptable.
  • Smoothness vs. Adaptation: Rapidly changing the video resolution to match fluctuating network bandwidth prevents stalls, but frequent quality switches are highly distracting to viewers.

On mobile hardware and wireless networks, these constraints are compounded by limited throughput and device power limitations, as discussed by Yang Du et al..


Formulating a Holistic QoE Metric

To navigate these trade-offs, systems require a unified mathematical framework. Instead of treating buffer management and bitrate selection as separate problems, multi-objective optimization combines them into a single formulation.

As demonstrated by Chao Zhou et al. with their QARC framework, a holistic QoE metric that explicitly balances perceived video quality, latency, and smoothness provides a significantly more effective target for optimization than single-factor metrics.

A typical multi-objective QoE formulation can be expressed conceptually as:

$$\text{QoE} = \sum (\text{Perceived Quality}) - \alpha \cdot (\text{Stall Duration}) - \beta \cdot (\text{Quality Variations}) - \gamma \cdot (\text{Latency})$$

Here, $\alpha$, $\beta$, and $\gamma$ act as weights that penalize negative experiences. By tuning these weights, the system can find an optimal operating point on the Pareto frontier—the set of choices where you cannot improve one metric without worsening another.

For instance, Yao Zhao 0001 et al. developed a multi-factor model for multiview streaming that integrates average quality, spatial-temporal variations, and starvation probability. By unifying these factors, their system can calculate a mathematically ideal buffer length that balances smooth playback with high visual fidelity, rather than relying on arbitrary, range-based buffering rules.


Context-Aware and Preference-Driven Trade-offs

The "ideal" balance of QoE objectives is not static; it changes based on the application, the network environment, and even the individual user.

Application and Network Constraints

In vehicular networks, fast-moving environments introduce severe volatility. Joongheon Kim et al. address this by engineering a multi-objective reward function that balances average video quality and stalls while minimizing packet drops at roadside base stations. Similarly, when video is delivered via Unmanned Aerial Vehicles (UAVs), network performance must be balanced against physical constraints; Kesong Wu et al. demonstrated that incorporating energy efficiency into the joint optimization framework can yield massive power savings while maintaining high user QoE.

User Preferences and Financial Costs

Sometimes the trade-off isn't just technical, but financial. In multi-path streaming environments (like using WiFi and cellular data simultaneously), maximizing resolution can lead to expensive cellular data overages. Ke Liu et al. designed a preference-aware system using Scalable Video Coding (SVC) that optimizes video quality while respecting user-defined constraints on cellular data costs. Taking personalization a step further, Imen Triki et al. introduced a closed-loop system that dynamically learns individual user preferences from direct feedback, tailoring the QoE weights to what a specific viewer values most.

Interactive and Emerging Media

New media formats introduce entirely new dimensions to the optimization problem. In social TV applications, where users watch content together and interact in real time, Weiwen Zhang showed that frameworks must jointly optimize video quality, resource efficiency, and inter-user delay to prevent one user's stream from lagging behind their friends' streams. For omnidirectional and 360-degree video, Xavier Corbillon et al. highlighted that algorithms must balance viewpoint switching lag alongside traditional resolution drops and playout stalls.


Solving the Optimization in Real Time

Formulating a multi-objective equation is only half the battle; the streaming client must solve it in real time under rapidly changing network conditions. Because a decision made right now affects the buffer state in the future, this is fundamentally a sequential decision-making problem under uncertainty.

To solve this, researchers frequently model the streaming environment as a Markov Decision Process (MDP) or use Deep Reinforcement Learning (DRL).

 [ Current State ] ---> ( DRL Agent / MDP Policy ) ---> [ Action Chosen ]
 (Buffer, Bandwidth)                                   (Bitrate, Node Selection)
         ^                                                        |
         |                                                        v
         +-------------- [ Multi-Objective Reward ] --------------+
                         (Quality - Stalls - Latency)

By framing the multi-objective QoE function as the reward function of an MDP, the streaming agent naturally learns a policy that maximizes long-term satisfaction.

  • Nesrine Changuel et al. utilized an MDP to schedule scalable video packets, balancing the maximization of video quality against the prevention of sender/receiver buffer violations.
  • Albert No et al. combined Lyapunov optimization with an MDP to handle a dual-timescale problem: coordinating network node selection over longer intervals while adjusting video quality levels in real time to prevent stalls.
  • In peer-to-peer and hybrid CDN architectures, Zhejiayu Ma et al. demonstrated that machine learning models can optimize peer selection by simultaneously evaluating throughput, latency, and chunk loss rates, moving beyond simplistic location-based heuristics.

Ultimately, multi-objective QoE optimization works because it acknowledges that "high-quality streaming" is not a single metric. By translating the complex, psychological trade-offs of human viewers into unified mathematical rewards, modern streaming systems can navigate volatile network landscapes to deliver the smoothest, clearest experience possible.

Key Foundational Work

Enhancing user experience for multi-screen social TV streaming over wireless networks (2014) — contributed 3 concept elements to this direction.
End-to-end stochastic scheduling of scalable video overtime-varying channels (2010) — contributed 2 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_multi_objective_qoe_optimization_5639f0e9,
  title = {multi-objective QoE optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-objective-qoe-optimization/rdr_5639f0e984a44f299ebd0d18eb2ec1f6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_multi_objective_qoe_optimization_5639f0e9_v1,
  title = {multi-objective QoE optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-objective-qoe-optimization/rdr_5639f0e984a44f299ebd0d18eb2ec1f6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake