Joint Quantization and Dynamic Partitioning for Context-Aware Edge Vision Transformers

Zhu Jiaqi, Jain Prakriti, Katare Dewant, Kumar SaurabharXiv 2026 · arXiv

TL;DR

This paper introduces a framework that jointly optimizes model partitioning and per-layer quantization for Vision Transformers (ViTs) in vehicular edge computing. By formulating the problem as a Mixed-Integer Linear Program (MILP), the authors significantly reduce energy consumption and latency for multi-stream perception tasks while maintaining high 3D detection accuracy.

What Problem Does This Paper Address?

Connected and Automated Vehicles (CAVs) rely on complex Vision Transformer models for situational awareness. However, these models are computationally expensive and memory-intensive, often exceeding the capabilities of local on-vehicle hardware.

While "split inference"—executing the first few layers on the vehicle and offloading the rest to a more powerful edge aggregator—is a known solution, existing methods are often static. They fail to adapt to dynamic conditions like fluctuating V2V (Vehicle-to-Vehicle) bandwidth, varying processing loads, or multiple concurrent sensor streams. Furthermore, most research treats model quantization (reducing numerical precision to save space) and partitioning as separate problems. This paper argues that because quantization directly changes the size of the data being transmitted, these two decisions must be made together to achieve true efficiency.

Methods

The authors propose a joint optimization framework called Joint DynOpt. Its core is a Mixed-Integer Linear Program (MILP) that minimizes the total energy (compute + communication) across all concurrent streams.

Key technical components include:

  • Joint Decision Space: The optimizer simultaneously selects the partition point for each stream, the node assignment (local vs. aggregator), and the per-layer precision (INT8 vs. FP16).
  • Exact Linearization: To keep the MILP solvable in real-time, the authors use auxiliary binary variables and "McCormick Envelopes" to linearize the coupled relationships between precision, activation size, and partition points.
  • Context-Awareness: The model accounts for profiled layer latencies that change based on the number of concurrent streams ($K$) and stream-specific V2V link rates.
  • Quality Constraints: Instead of just maximizing accuracy, the framework treats task quality as a constraint, using an empirically calibrated "sensitivity budget" to ensure the final 3D detection performance remains acceptable.
Fig. 1: System model architecture and split-inference pipeline across local on-vehicle platforms (nlocal ∈{nNano, nNX}) and the AGX Orin aggregator (nAGX). The optimization coordinator executes on the aggregator node, taking telemetry feedback and channel conditions to issue optimal partition points (zl,k), precision allocations (yl,b), and node assignments (xl,n,k).

Datasets

The framework was evaluated using the OPV2V benchmark, a large-scale dataset for V2V collaborative perception.

  • Scenarios: The evaluation used LiDAR and camera perception traces from CARLA (simulated) and Culver City (real-world) environments.
  • Model: A 12-layer Vision Transformer backbone was used as the primary computational workload.
  • Hardware Testbed: A heterogeneous setup featuring NVIDIA Jetson Orin Nano and Orin NX as local nodes, and a Jetson AGX Orin as the edge aggregator.

Key Insights and Ideas

  • The Quantization-Communication Link: The "aha moment" here is that quantization isn't just about faster math; it's a compression tool for the network. Selecting INT8 for a layer right before a split point can halve the transmission energy for that stream.
  • Non-Monotonic Sensitivity: The authors found that quantization sensitivity isn't uniform across a Transformer. Some layers (like 1, 6, 9, 11, and 12) are highly sensitive and should stay in FP16, while others can be aggressively quantized to INT8 without hurting the final mAP (mean Average Precision) significantly.
  • Real-Time Feasibility: Despite the complexity of MILP, the solver (SCIP 8.0) consistently found optimal solutions in under 10ms on the Jetson AGX Orin, making it suitable for the high-speed requirements of vehicular networking.
Fig. 2: Effect of layer-wise mixed-precision selection on quantization sensitivity and intermediate activation size.

Key Results

The Joint DynOpt framework was compared against several baselines, including full local execution and uniform INT8 splits.

  • Energy Savings: The proposed method reduced V2V communication energy by 33.3–36.0% compared to a uniform INT8 split and by up to 70.1% compared to static FP16 splits.
  • Latency Reduction: At a high workload of 8 concurrent streams ($K=8$), end-to-end latency dropped by 34.0% on the Orin Nano and 33.1% on the Orin NX relative to the uniform INT8 baseline.
  • Accuracy Trade-off: The framework limited the drop in 3D detection accuracy to just 3.1–4.1 percentage points (mAP50) compared to the "gold standard" full FP16 execution, while being significantly faster and more energy-efficient.

Limitations and Future Work

The authors acknowledge a few areas for expansion:

  • Precision Granularity: The current work focuses on INT8 and FP16. Future iterations could explore sub-8-bit quantization (e.g., INT4) for even greater savings.
  • Semantic Communication: The paper suggests that further gains could be made by incorporating "token reduction" techniques—identifying and discarding less important visual tokens before transmission.
  • Dynamic Link Conditions: While the model handles different link rates, testing under more extreme mobility-induced fading or interference would further validate its robustness in real-world V2X scenarios.

Summary updated

Abstract

Deploying vision transformer models across connected and automated vehicles (CAVs) requires joint management of computation and Vehicle-to-Vehicle (V2V) communication. Static split-inference methods use fixed partition points and cannot adapt to time-varying link bandwidth, processing load, and concurrent multi-stream traffic. This paper proposes an edge orchestration framework that jointly selects model partition points, per-layer quantization bitwidths, and execution placement across heterogeneous vehicular compute nodes. The joint partitioning and mixed-precision allocation problem is formulated as a Mixed-Integer Linear Program (MILP) that minimizes compute and communication energy subject to end-to-end latency and task-quality constraints. The framework is evaluated on Jetson Orin Nano (8~GB), Jetson Orin NX (16~GB), and Jetson AGX Orin (32~GB) nodes using OPV2V-driven multi-stream workloads with $K \in \{2,4,6,8\}$. Across the evaluated configurations, the proposed formulation reduces V2V communication energy by $33.3$--$36.0\%$ relative to a Uniform INT8 split. At $K=8$, end-to-end task completion latency is reduced by $34.0\%$ on the Orin Nano and $33.1\%$ on the Orin NX relative to the same baseline, while limiting the reduction in 3D detection accuracy to $3.1$--$4.1$ percentage points in $\mathrm{mAP}_{50}$ relative to FP16 execution.
BibTeX
@misc{jiaqi2026joint,
  author = {Zhu Jiaqi and Jain Prakriti and Katare Dewant and Kumar Saurabh},
  title = {{Joint Quantization and Dynamic Partitioning for Context-Aware Edge Vision Transformers}},
  year = {2026},
  eprint = {2611.14196},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14196},
}
Figure