When someone watches video on a phone on a moving bus, the screen shakes relative to their eyes. That shaking smears fine detail, so a 1080p chunk may look no better than a 540p one, and the extra bytes are wasted. Environment-aware streaming has exploited this effect only coarsely. The leading system sorts the viewer's surroundings into three classes ("quiet and static", "normal", "dynamic and noisy") by thresholding accelerometer and microphone readings, then reweights a standard bitrate controller for each class (Enhancing QoE for Mobile Users by Environment-Aware HTTP Adaptive Streaming). We propose to replace that hand-set class lookup with a physical model of how much detail the viewer can actually see. The model maps the phone's measured motion to a blur kernel. Its few free parameters are fitted against controlled human experiments on a vibration platform, in the same simulate–measure–recalibrate loop that engineering uses to validate finite-element models. The client then uses the calibrated model only to drop bitrate rungs that the viewer cannot distinguish from the top rung. It leaves the stall-avoidance logic of a proven buffer controller untouched.
The opportunity matters because the existing evidence mixes two effects. In the dynamic and noisy setting, the environment-aware controller cut stalls from 13 to 4 and raised mean opinion score (MOS, the average 1–5 viewer rating) from 3.09 to 3.79 over plain BOLA (Enhancing QoE for Mobile Users by Environment-Aware HTTP Adaptive Streaming). Buses also have worse networks, though. Nobody has shown how much of that gain comes from perception, where shaking hides detail, and how much comes from simply being more cautious on a bad link. Our design separates the two. It holds the network fixed while varying vibration in the lab, and the result tells us whether sensor context is worth building into players at all.
Core Claim
Perceived quality under hand-held vibration can be predicted by blurring both the reference and the compressed frame with a motion kernel derived from the phone's inertial sensors, then scoring the pair with VMAF, a learned full-reference quality metric. Two calibrated parameters are needed: an effective viewing distance, and a gain that accounts for the eye partly tracking the shaking screen. With those, this "effective quality" should rank human pairwise preferences across bitrate rungs better than raw VMAF and better than any three-class rule. Pruning the bitrate ladder with this measure should cut bytes in shaky conditions at no measurable MOS loss, even when the network trace is held identical.
The Model and Its Calibration
From the accelerometer and gyroscope, we estimate the screen's displacement over each frame's display interval. We convert that displacement to a visual angle using a viewing distance $d$, and scale it by a tracking gain $g \in [0,1]$, where $g=1$ means the eye does not follow the screen at all. This gives a per-chunk blur kernel $K_{d,g}$. Its extent in pixels is $g\,\Delta x / (d\,\phi)$, where $\Delta x$ is screen displacement and $\phi$ is the angular size of one pixel. For each rung $r$, effective quality is
$$Q_{\text{eff}}(r) = \text{VMAF}\big(K_{d,g} * \text{ref},\; K_{d,g} * \text{dist}_r\big),$$
where $*$ is convolution, ref is the source frame, and $\text{dist}_r$ is the decoded rung-$r$ frame upscaled to screen resolution.
The uncalibrated version, with nominal $d$ and $g=1$, is the analogue of an idealized finite-element model, and we expect it to be wrong in a specific direction. Smooth-pursuit eye movements and head–hand coupling mean real viewers see less blur than pure screen motion implies. The calibration step mirrors the bellows study (Mechanism of Mechanical Analysis on Torsional Buckling of U-Shaped Bellows in FLNG Cryogenic Hoses). There, an idealized simulation erred by 3.3%, and fitting one measured physical deviation, wall thinning, brought the error to 0.9%. Here we fit $(d, g)$ to human pairwise judgments with a Bradley–Terry likelihood, the same model the MSU benchmark uses to turn pairwise votes into scores (Video compression dataset and benchmark of learning-based video-quality metrics). We expect to need only a handful of parameters, which keeps calibration feasible with tens of participants rather than thousands. The physical-specimen papers warn that tiny samples limit what can be fitted (Experimental and Numerical Study of Static Behavior of Precast Segmental Hollow Bridge Piers), so we deliberately keep the parameter count low.
Research Program
The work has three parts.
The first is a vibration rig. A phone and a seated or standing viewer are placed on a motion platform that replays recorded inertial traces from walking, buses, and trains at several amplitudes, plus a still condition. Participants make pairwise comparisons between rungs of the same clip.
The second is the controller. Our integration choice is constrained by the Puffer deployment of BOLA with SSIM (Implementing BOLA-BASIC on Puffer: Lessons for the use of SSIM in ABR logic). In that study, swapping a perceptual metric directly into BOLA's utility broke the monotone link between chunk size and utility and drained buffers. We therefore never feed $Q_{\text{eff}}$ into the controller's objective. Instead, before each decision, we remove from the ladder every rung above $r^\star$, the cheapest rung whose effective quality is close enough to the top rung:
$$r^\star = \min\{\, r : Q_{\text{eff}}(r) \ge Q_{\text{eff}}(r_{\max}) - \delta \,\},$$
where $\delta$ is a just-noticeable difference measured in the rig. Unmodified BOLA (BOLA: Near-Optimal Bitrate Adaptation for Online Videos) then runs on the truncated ladder. BOLA's utility stays monotone in chunk size on that subset, so its near-optimality argument still applies to the remaining rungs.
The third is a cost audit. We measure the device-side energy of continuous sensor polling and blur computation, which the class-based work never reported. We precompute blurred VMAF offline for a grid of kernel extents, so the client only does a table lookup. We also measure whether pruning 4K or 1080p rungs lowers decoding energy, in line with the finding that energy-optimal ladders favor lower resolutions (Rate-Quality or Energy-Quality Pareto Fronts for Adaptive Video Streaming?).
Baselines
The naive reference is plain BOLA on the full ladder, with no sensors. It is the standard against which the environment-aware work was measured, and it is reported within 84–95% of an offline optimum on 3G traces (BOLA: Near-Optimal Bitrate Adaptation for Online Videos).
The strongest existing method is the three-class environment-aware BOLA of Enhancing QoE for Mobile Users by Environment-Aware HTTP Adaptive Streaming. It reports MOS 3.79 vs. 3.09 for BOLA in dynamic and noisy settings, and 4.05 vs. 3.79 in quiet and static ones. We will reimplement it from its description, with the same accelerometer and microphone thresholds, the same videos, the same network traces, and the same player. Its thresholds and weights get the same tuning budget on our rig data that our $(d, g, \delta)$ get. We beat it if we deliver equal or higher MOS with fewer bytes under vibration, or higher MOS at equal bytes. The comparison runs both on fixed traces and in the field.
A third baseline is a quality-aware learned controller such as Comyco, which optimizes chunk-level VMAF without context. It reports 7.5–16.79% higher QoE than BOLA, MPC, and Pensieve in trace emulation (Comyco: Quality-Aware Adaptive Video Streaming via Imitation Learning). This tests whether better use of raw VMAF already captures the gain.
Two ablations isolate the key ingredient. The first, "uncalibrated physics", uses $g=1$ and a nominal $d$, and tests whether calibration matters. The second, "three-class pruning", applies our ladder truncation with only one $\delta$ per Zhang-style class, and tests whether a continuous kernel matters beyond coarse classes.
Why not just…?
Why not just keep the three-class heuristic, since it already works? This is the most serious threat. If vibration in the real world clusters into a few regimes, three classes may capture most of the perceptual gain. The three-class-pruning ablation settles it directly. If continuous calibrated blur predicts held-out pairwise preferences no better than class membership, and saves no more bytes at equal MOS, we will report that and recommend the simpler rule. We expect a difference at intermediate amplitudes such as walking and smooth trains, which the class thresholds lump together.
Why not just make the player more conservative on cellular links? A larger buffer target or a bitrate cap on mobile data could reproduce the stall reduction reported for buses without any sensing, because much of that gain may be a network effect. This is exactly why the lab phase replays identical network traces (the 3G traces used for BOLA, and the FCC and Norway traces used by Mowgli: Passively Learned Rate Control for Real-Time Video) across vibration conditions. Any byte saving we measure there cannot come from the network. A conservative-cap baseline, tuned to match our bytes, will also be run. If it matches our MOS, the perceptual mechanism is not doing work.
Why not just learn context weights from engagement logs, as LingXi does with exit rates? Towards User-level QoE: Large-scale Practice in Personalized Optimization of Adaptive Video Streaming personalizes controller parameters from exit rates across tens of millions of users, but its total gains are small (0.15% viewing time, 1.3% stall time). It also requires live online exploration and production telemetry that we do not have. Exit rates in a bus are confounded by people reaching their stop, which a vibration-to-exit regression cannot separate from blur. A controlled rig gives clean causal calibration with few participants. Logs remain a natural later validation once sensor-tagged telemetry exists.
Proposed Experiments
The source clips come from the Comyco DASH set, which has chunk-level VMAF across bitrates for 86 videos (Comyco: Quality-Aware Adaptive Video Streaming via Imitation Learning), plus the HEVC YouTube-UGC subset at 720p, 1080p, and 2160p from Rate-Quality or Energy-Quality Pareto Fronts for Adaptive Video Streaming?. Together they cover both ordinary ladders and 4K.
The first decisive check is predictive. On rig sessions held out by vibration trace and by participant, we compare $Q_{\text{eff}}$ with calibrated $(d,g)$ against raw VMAF, uncalibrated blur, and class membership on pairwise-preference accuracy and Spearman correlation with Bradley–Terry scores. The hypothesis fails if calibrated blur does not beat raw VMAF under shaking while matching it when still. For reference, raw VMAF reaches SROCC 0.943 on the static MSU benchmark (Video compression dataset and benchmark of learning-based video-quality metrics). No reported result was found on Lacuna for any quality metric under physical vibration.
The second check holds the network fixed. Under identical replayed traces, we compare bytes downloaded, stall count, MOS, and decoding energy across all baselines at each vibration level. The prediction is a byte saving that grows with vibration amplitude and is zero when still, at MOS within the measured just-noticeable difference of full-ladder BOLA.
The third check is a field replication across library, campus, home, and bus, mirroring the four settings used in Enhancing QoE for Mobile Users by Environment-Aware HTTP Adaptive Streaming. There we compare MOS and stalls directly against its reported 3.79 and 4.05 figures, and report the sensing energy overhead it left unmeasured.
Two issues are out of scope. Ambient noise affects tolerance for stalls rather than visible detail; we leave it to the buffer controller and say so. Fast-motion content may already contain motion blur that interacts with shake blur, so we stratify results by content motion.
References
- Enhancing QoE for Mobile Users by Environment-Aware HTTP Adaptive Streaming
- Mechanism of Mechanical Analysis on Torsional Buckling of U-Shaped Bellows in FLNG Cryogenic Hoses
- Video compression dataset and benchmark of learning-based video-quality metrics
- Experimental and Numerical Study of Static Behavior of Precast Segmental Hollow Bridge Piers
- Implementing BOLA-BASIC on Puffer: Lessons for the use of SSIM in ABR logic
- BOLA: Near-optimal bitrate adaptation for online videos
- Rate-Quality or Energy-Quality Pareto Fronts for Adaptive Video Streaming?
- Comyco: Quality-Aware Adaptive Video Streaming via Imitation Learning
- Mowgli: Passively Learned Rate Control for Real-Time Video
- Towards User-level QoE: Large-scale Practice in Personalized Optimization of Adaptive Video Streaming