Training Bitrate Controllers for the Rating Viewers Remember

A video player chooses the quality of each next few seconds of video from a menu of encodings. Higher quality looks better, but if the network cannot deliver it in time the video freezes. Learned controllers of this kind, trained by reinforcement learning (trial-and-error optimization of a reward), score themselves with a per-chunk sum: reward for quality, a penalty for each freeze, and a penalty for every quality switch, with later chunks discounted. Viewers do not score a session that way. Their satisfaction drops at once when quality degrades and recovers only slowly. Their overall judgment is dominated by the worst moment and by whatever happened near the end. So a controller can do well on its training reward while producing sessions people remember as bad.

The proposal is to train the controller against a model of the viewer's memory rather than a per-chunk sum. The controller would carry a small estimate of the viewer's current remembered satisfaction as part of what it observes. It would be rewarded by how much each decision changes the rating the viewer would give if the session ended now. Its usual time-discount would be replaced by the probability that the viewer is still watching. In plain terms, the controller should protect what the viewer will remember at the unknown moment they stop watching. The central test is whether such a controller beats carefully tuned conventional controllers when judged by a quality-of-experience model it was not trained on, and by human ratings.

Core Claim

Several papers show that a viewer's overall rating of a session is not additive over time. Modeling Continuous Video QoE Evolution: A State Space Approach fits a third-order nonlinear state-space model: per-moment quality, playback state and time since the last freeze drive a hidden "memory" state. It tracks continuous satisfaction with LCC 0.83 and SROCC 0.79 on the LFOVIA database. Video ATLAS (Learning to Predict Streaming Video QoE: Distortions, Rebuffering and Memory; Towards Perceptually Optimized End-to-end Adaptive Video Streaming) predicts end-of-session opinion scores from VMAF quality, stall features and a time-since-impairment memory feature. It reaches SROCC 0.875 on LIVE-Netflix and 0.87 on LIVE-NFLX-II, and it documents that stall dissatisfaction plateaus beyond about 4 seconds. The authors of the second paper explicitly call for, but do not build, a closed loop in which such memory models drive the controller.

Learned controllers ignore these effects. Neural Adaptive Video Streaming with Pensieve trains a policy on a linear per-chunk reward and reports 12–25% average gains over heuristics. QARC adds a network that predicts upcoming VMAF scores but still penalizes switches symmetrically and discounts geometrically. The nearest work that does use perceptual memory for rate control is Rate Adaptation and Admission Control for Video Transmission with Subjective Quality Constraints. It uses a cumulative "area of disappointment" statistic, which correlates with ratings at LCC 0.8446 versus 0.5659 for mean quality, inside a convex Lyapunov scheduler at the base station. That statistic never forgets and does not saturate. It also offers no answer to the question of when the rating is taken.

The claim is that these pieces combine into one well-posed objective, with three parts. First, let $Q_t$ be a memory model's predicted overall rating if the session ended after chunk $t$. Second, let $m_t$ be that model's internal state: the state-space memory vector plus a running worst-quality value, so that the negative-peak effect stays Markov. Third, let $\tau$ be the random chunk at which the viewer stops. The controller should maximize $\mathbb{E}[Q_\tau]$. Giving the reward $r_t = Q_t - Q_{t-1}$ makes the rewards telescope:

$$\mathbb{E}[Q_\tau] = Q_0 + \sum_{t \ge 1} S(t)\, \mathbb{E}[r_t], \qquad S(t) = P(\tau \ge t).$$

Here $S(t)$ is the viewer survival curve. This resolves a real tension in the material. Geometric discounting, as recommended for stable long-horizon learning by IntentNet, devalues exactly the late window that recency bias says matters most. Hand-set end-ramps, as in BiLSTM-ST and RSBG, assume a known end. The survival weight $S(t)$ is neither choice. It is the per-step weight implied by an unknown end. Recency is then handled by the memory model itself rather than by the discount. The earliness literature supplies the framing that the payoff is read off at a stopping time (Deep Recurrent Optimal Stopping), except that here the viewer, not the agent, does the stopping.

A caveat belongs up front. The reward transformation does not change the optimal policy relative to paying $Q_\tau$ once at the end. It only makes credit assignment denser. Any gain from the method must therefore come from two other ingredients: putting $m_t$ in the controller's observation, and weighting by survival instead of an arbitrary discount. The experiments are designed around that.

Research Program

The work has three parts.

Building the in-loop memory model. We fit the state-space model of Eswara et al. on LFOVIA continuous traces and extend it in two ways. We add a running minimum to capture the negative-peak effect. We add a saturating stall input so that it matches the roughly 4-second duration-neglect plateau reported on LIVE-NFLX-II. Its output $Q_t$ is a calibrated "rating if stopped now."

Training the controller. We train a Pensieve-style actor-critic controller in a trace-driven chunk simulator. Its observation is the standard Pensieve state augmented with $m_t$, its reward is $r_t$, and its return weights come from a parametric survival curve. We train across a sweep of survival curves, from fixed-length 40-second sessions to heavy-tailed abandonment. The file contains no measured abandonment curves, so these are explicit assumptions.

Analyzing behavior. The theory makes qualitative predictions that a symmetric penalty cannot produce. The new controller should be reluctant to make up-switches it cannot sustain, because a short climb buys little remembered quality while the later drop costs a lot. It should spend buffer risk early, when the memory model forgets, and protect the likely final window. We will measure the up/down switch ratio and the timing of freezes relative to session end.

Baselines

The naive reference point is BOLA (BOLA: Near-Optimal Bitrate Adaptation for Online Videos), a buffer-based controller with no learning. It is run on identical traces and videos with its standard utility settings. Failing to beat it clearly would end the project.

The strongest existing method is Pensieve with its linear quality, freeze and smoothness reward. Pensieve reports 12–25% average QoE gains over heuristics on its own linear metric. There is no reported result found on Lacuna for Pensieve, QARC or BOLA evaluated by a retrospective memory-based QoE model on LIVE-NFLX-II content, so all such numbers will be produced here.

To be fair to Pensieve, we will give it its strongest form. It gets separate up-switch and down-switch penalties, a separate freeze weight, and a tuned discount. Its hyperparameters are searched under the same budget of training runs, simulator steps and network size as our method. Every variant is selected on a validation evaluator, never the test evaluator.

The terminal-reward agent is the most important competitor. It receives the memory model's score once, at the end of the session, with no discount and no memory state in its observation. As argued above, it targets the same optimum.

Two ablations isolate our key ingredients. The first removes $m_t$ from the observation but keeps the shaped reward and survival weights. The second keeps $m_t$ but replaces $S(t)$ with geometric discounting.

We count the method as beating a baseline only if three things hold. It must improve the held-out evaluator's predicted mean opinion score with confidence intervals over trace sets that exclude zero. It must improve the fraction of sessions above a dissatisfaction threshold, the satisfaction-rate metric used by Chen et al. And it must win the human-rating comparison described below.

Why not just…?

Why not just tune Pensieve's reward with asymmetric switch penalties and a heavier late-session weight? This is a real threat, because much of the hysteresis effect might be captured by penalizing down-switches more than up-switches. A fixed per-switch penalty cannot express that a drop's cost depends on how long ago the last drop was, or that the worst moment so far changes the value of everything after it. Both are effects the state-space and ATLAS memory features exist to model. Rather than argue the point, we include the tuned asymmetric Pensieve as a baseline with an equal search budget. If it matches us on held-out evaluators and human ratings, the conclusion is that hysteresis is reward tuning and not state, and we will report that.

Why not just put Video ATLAS's end-of-session score in as a terminal reward? In expectation this targets the same objective, so the honest question is whether dense shaping, the memory state and survival weighting matter in practice. The terminal-reward agent is a baseline, and the two ablations separate learning speed from final quality. If the terminal agent catches up given enough samples, the contribution shrinks to sample efficiency, which is still relevant given QARC's report of up to 22 hours of training with 8 agents.

Why not just apply a hysteresis filter to an existing controller's output, or use the cumulative disappointment statistic? A post-hoc filter only measures sessions; it cannot change decisions. Chen et al.'s statistic does change decisions, but it never forgets past disappointment and does not saturate on long stalls, which conflicts with the duration-neglect finding. We will run a variant whose memory state is that cumulative statistic to test whether forgetting and saturation matter.

Proposed Experiments

The simulator replays the FCC and Norway throughput traces used by QARC through LIVE-NFLX-II contents encoded at a fixed set of bitrates, with per-chunk VMAF scores.

The decisive check guards against a controller that games its own reward model. Controllers whose memory model was fitted on LFOVIA are evaluated with a Video ATLAS model retrained on LIVE-NFLX-II, and the reverse. Gains that appear only under the controller's own model count as failures.

We then render matched sessions from each controller under identical traces and collect retrospective ratings with the LIVE-NFLX-II protocol. This includes a set of longer clips, because the file notes that long-session hysteresis is largely unmeasured. A paired comparison of our method against tuned Pensieve on these ratings is the result that settles the claim.

Secondary checks test the method's own predictions. First, the predicted asymmetries should actually appear: fewer unsustained up-switches, and freezes shifted away from the likely session end. Second, survival weighting should outperform geometric discounting most when session lengths are heavy-tailed and least when they are fixed. If it wins equally everywhere, the gain is not coming from the mechanism we propose.