RESA: Recurrent Feature-Shift Aggregator for Lane Detection

Tu Zheng, Hao Fang, Yi Zhang, Wenjian Tang, Zheng Yang, Haifeng Liu, Deng Cai

Introduction

Lane detection is a fundamental task for autonomous driving, yet it remains challenging due to the inherent characteristics of lanes. Lanes are typically long, thin structures that are frequently subject to severe occlusion by other vehicles, poor lighting conditions, or degraded road markings. Standard Convolutional Neural Networks (CNNs) are highly effective at capturing local patterns but often lack the global receptive field necessary to "infer" a lane's path when visual cues are missing.

Previous state-of-the-art methods, such as Spatial CNN (SCNN), addressed this by treating feature maps as sequences and passing information pixel-by-pixel. While this approach captures global context, its sequential nature is computationally expensive and slow, making it unsuitable for real-time applications. The Recurrent Feature-Shift Aggregator (RESA) addresses these limitations by introducing a parallelized, log-scale shift operation that gathers global spatial information both effectively and efficiently.

Methodology: The RESA Framework

The core innovation of this work is the Recurrent Feature-Shift Aggregator (RESA). Instead of the sequential message passing used in prior models, RESA slices the feature map and shifts these slices across the image in four directions: up, down, left, and right.

Recurrent Feature-Shift Aggregation

In each iteration $k$, feature slices are shifted by a dynamic stride $s_k = L / 2^{K-k}$, where $L$ is the dimension of the feature map and $K$ is the total number of iterations. This logarithmic stride ensures that every pixel can aggregate information from every other pixel in just a few steps (typically $K=4$ or $5$). Because all slices are updated simultaneously, the operation is highly optimized for GPU parallelism.

Overview of the RESA information aggregation process.

Bilateral Up-Sampling Decoder (BUSD)

To complement the aggregator, the authors propose the Bilateral Up-Sampling Decoder (BUSD). Standard decoders often rely on simple bilinear interpolation, which can lose fine-grained spatial details. BUSD employs two branches:

  1. Coarse-grained branch: Utilizes bilinear up-sampling to capture the overall scene structure.
  2. Fine-detailed branch: Uses transpose convolutions and "non-bottleneck" blocks to recover precise lane boundaries and subtle edges.

Experimental Evaluation

RESA was evaluated on two major lane detection benchmarks:

  • CULane: A large-scale dataset with over 133,000 frames covering challenging scenarios like night driving, crowded roads, and missing markings.
  • TuSimple: A dataset focused on highway scenes, used to measure high-accuracy lane marking.

Key Results

RESA achieves a state-of-the-art balance between accuracy and speed:

  • Performance: On CULane, RESA-50 achieved an F1-measure of 75.3, surpassing previous leaders like SCNN and SAD. On TuSimple, it reached an accuracy of 96.82%.
  • Efficiency: The RESA module is significantly faster than its predecessors. While SCNN's message-passing takes approximately 43.5ms, the RESA aggregator requires only 3.8ms—a 10x speedup for that specific component.
  • Robustness: Qualitative analysis demonstrates that RESA maintains smooth, continuous lane predictions even in "No Line" or "Shadow" scenarios where standard CNNs often produce fragmented or noisy results.
Qualitative comparison between standard CNN and RESA.

Conclusion

RESA introduces a highly efficient mechanism for global information aggregation in lane detection. By leveraging logarithmic shifts, it achieves the necessary "conjecture" capability—predicting lane paths based on road geometry when markings are occluded—without the computational burden of sequential models. Its modular, "plug-and-play" design allows it to be easily integrated into existing architectures, making it a powerful tool for real-time autonomous perception.

Summary updated

Abstract

Lane detection is one of the most important tasks in self-driving. Due to various complex scenarios (e.g., severe occlusion, ambiguous lanes, etc.) and the sparse supervisory signals inherent in lane annotations, lane detection task is still challenging. Thus, it is difficult for the ordinary convolutional neural network (CNN) to train in general scenes to catch subtle lane feature from the raw image. In this paper, we present a novel module named REcurrent Feature-Shift Aggregator (RESA) to enrich lane feature after preliminary feature extraction with an ordinary CNN. RESA takes advantage of strong shape priors of lanes and captures spatial relationships of pixels across rows and columns. It shifts sliced feature map recurrently in vertical and horizontal directions and enables each pixel to gather global information. RESA can conjecture lanes accurately in challenging scenarios with weak appearance clues by aggregating sliced feature map. Moreover, we propose a Bilateral Up-Sampling Decoder that combines coarse-grained and fine-detailed features in the up-sampling stage. It can recover the low-resolution feature map into pixel-wise prediction meticulously. Our method achieves state-of-the-art results on two popular lane detection benchmarks (CULane and Tusimple). Code has been made available at: https://github.com/ZJULearning/resa.
BibTeX
@inproceedings{zheng2021resa,
  author = {Tu Zheng and Hao Fang and Yi Zhang and Wenjian Tang and Zheng Yang and Haifeng Liu and Deng Cai},
  title = {{RESA: Recurrent Feature-Shift Aggregator for Lane Detection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i4.16469},
}
Figure