Autonomous systems operating in the real world—such as self-driving vehicles, drones, and wearable robotics—must continuously perceive their surroundings and forecast future events. To do this safely, they rely on complex deep learning models for tasks like semantic segmentation and trajectory prediction. However, these systems operate under unforgiving physical constraints.
As Apoorv Singh notes, onboard hardware must execute these models within strict latency budgets, typically around 10Hz, while processing high-resolution data streams. Izzeddin Teeti et al. emphasize that this creates a severe gap between research-grade accuracy and practical onboard utility. If a model takes too long to compute a pixel mask or predict a pedestrian's path, the vehicle cannot react in time.
The core challenge of this research direction is breaking the accuracy-latency trade-off: how can we achieve high-fidelity spatial and temporal reasoning without exceeding the millisecond-level constraints of edge hardware?
Streamlining the Representation Space
One of the most intuitive ways to reduce latency is to rethink what the model is trying to predict. Traditional computer vision pipelines often use complex multi-stage architectures to output highly detailed polygons or instance masks. However, this level of detail is often computationally redundant for downstream planning tasks.
For instance, Hu Cao et al. demonstrate this by simplifying parking slot detection. Instead of treating slots as complex polygons that require heavy geometric reasoning, they frame them as simple "directional entrance lines." This representation shift allows a streamlined 30-layer network to achieve a 13ms processing speed without sacrificing precision.
Similarly, in trajectory forecasting, Haicheng Liao et al. eliminate the need for computationally expensive High-Definition (HD) maps entirely. By treating traffic interactions as a dynamic social network and applying graph centrality measures, their framework achieves state-of-the-art accuracy with up to 90% fewer parameters than map-heavy alternatives.
Efficient Context Aggregation
To segment an image accurately, a model needs global context—it must understand how a pixel relates to the rest of the scene. Modern foundation models like the Segment Anything Model (SAM) excel at this, but as Alexey Nekrasov 0001 et al. point out, their massive size makes them impractical for real-time production systems. Chao Yao also notes that relying on dense instance-specific routing can increase inference latency by an order of magnitude.
To capture global context efficiently, researchers are replacing dense, quadratic-cost self-attention with specialized, lightweight alternatives. Kangfu Mei et al. introduce a Strip Attention Module (SAM) in their AttaNet architecture, which captures global context along horizontal and vertical strips rather than the entire dense grid. To handle the high-resolution demands of remote sensing, Zhen-Liang Ni et al. propose a Rectangular Self-Calibration Module (RCM) that drastically reduces GFLOPs compared to standard Vision Transformers.
Furthermore, instead of relying on slow, heavy decoders, Jie Hu et al. propose a "convolution-first" and separable design in their YOSO framework, bringing panoptic segmentation into the real-time era by optimizing the feature aggregation layer itself. The authors of CFPNet achieve a similar balance by using a Channel-wise Feature Pyramid, compressing spatial details into efficient channel allocations to maintain a tiny parameter footprint.
Injecting Physics and Heuristic Biases
Purely data-driven models often waste computational cycles trying to learn the basic laws of physics or geometry from scratch. By injecting explicit inductive biases, models can bypass heavy architectural layers.
In motion forecasting, Carlos Gómez Huélamo et al. combine deep neural networks with physics-based heuristic scene understanding, significantly reducing the model's parameter footprint. Taking this further, Chengyue Wang et al. introduce the DEMO framework, which couples a Dynamic Bicycle Model with a two-stage deep learning architecture. Because the bicycle model handles the rigid kinematics of vehicle motion, the neural network only needs to refine the strategic trajectory, resulting in a lightning-fast 15ms inference speed.
Similarly, Chen Feng 0006 et al. explicitly couple map constraints with agent motions via a bilateral query mechanism, forcing the network to focus only on physically plausible paths and achieving a 10x speedup over unconstrained transformers.
Collaborative Learning and Complexity Prioritization
When deploying on edge devices, we often face a dilemma: convolutional networks are fast but lack global context, while transformers are accurate but slow. Jinna Lv et al. highlight this fundamental trade-off in their remote sensing review.
To bridge this gap, the authors of a collaborative learning framework use online knowledge distillation to let CNNs and Vision Transformers co-train. This allows a compact, deployable CNN student to inherit the global reasoning capabilities of a heavy transformer teacher without carrying its computational weight to the edge hardware.
Another optimization strategy is to allocate compute dynamically based on scene difficulty. Lightweight networks often fail at complex boundaries or small objects, as noted by Genshun Dong et al. To solve this, Xin Zhang et al. introduce an image complexity prior that guides feature refinement. By predicting which regions of an image are hardest to segment, the network dynamically focuses its limited computational budget on fine edges and dense information zones, maximizing efficiency.
Hardware Aware Co Design
Ultimately, software optimizations must align with the underlying hardware. Peiwen Lin et al. address this by automating network design through Graph-guided Architecture Search (GAS), optimizing both pixel-level precision and latency directly within the search loop.
For aerial robotics, where power and weight constraints are extreme, Lei Sun et al. demonstrate a hardware-software co-design. By pairing a specialized Panoramic Annular Lens with a highly efficient network, they achieve 360° situational awareness on embedded drone hardware, proving that real-time performance is achieved when the physical sensor and the neural architecture are optimized in tandem.