NightLab: A Dual-level Architecture with Hardness Detection for Segmentation at Night

Xueqing Deng, Peng Wang, Xiaochen Lian, Shawn Newsam

TL;DR

NightLab is a novel dual-level framework designed to tackle the unique challenges of nighttime semantic segmentation by combining global image-level estimation with targeted regional refinement. By introducing a regularized light adaptation module (ReLAM) and a context-aware hardness detection module (HDM), NightLab effectively handles severe illumination variations and recovers fine-grained details for small or ambiguous objects, achieving state-of-the-art performance on nighttime benchmarks.

What Problem Does This Paper Address?

Semantic segmentation has seen massive improvements recently, but progress has been heavily biased toward daytime scenes. Driving at night introduces severe computer vision challenges: extreme illumination variations (such as glare from oncoming headlights or pitch-black underexposed areas), blurry textures, and low contrast.

Standard approaches often rely on Unsupervised Domain Adaptation (UDA) to bridge the gap between day and night, but the domain gap is frequently too wide for these frameworks to perform reliably in production. Furthermore, small or slim objects (like traffic poles or distant riders) lose their distinct visual features in the dark. Without proper surrounding context, modern deep networks easily misclassify them (e.g., confusing a rider on a motorcycle for a pedestrian walking).

Methods

NightLab addresses these issues using a dual-level (image-level and region-level) architecture that processes images in parallel pipelines:

Overview of the NightLab dual-level inference architecture.
  1. Image-Level Pipeline: The full nighttime image is first passed through the Regularized Light Adaptation Module (ReLAM) ($\Phi^I_{light}$). Unlike standard image-to-image translation methods (like CycleGAN) which can severely distort image textures, ReLAM uses structural similarity (SSIM) and GAN losses to adjust illumination while strictly preserving underlying textures. The enhanced image is then segmented by an image-level segmentation network ($\Phi^I_{seg}$) using a Swin-Transformer encoder and a UperNet decoder enhanced with Deformable Convolutions (DefConv) for superior context modeling.
  2. Hardness Detection Module (HDM): In parallel, the Hardness Detection Module ($\Phi^R_{det}$) analyzes the image to identify "hard regions"—areas where the global image-level model is likely to fail.
  3. Region-Level Pipeline: These hard regions are cropped, batched, and sent to a regional pipeline ($\Phi^R_{light}$ and $\Phi^R_{seg}$). Crucially, the regional segmentation network is only trained on a subset of difficult classes (like bicycles or rider) to specialize its feature extraction. Finally, these high-resolution regional predictions are merged back into the global prediction map to produce the final output.

Datasets

The authors evaluated NightLab on two primary nighttime datasets:

  • NightCity / NightCity+: NightCity is a large-scale dataset containing densely labeled nighttime urban driving scenes. The authors discovered significant pixel-level mislabeling in the original validation set, leading them to manually relabel it to create NightCity+ for a more accurate and meaningful evaluation.
  • BDD100K-Night: A subset of the BDD100K dataset consisting of real-world nighttime driving images (343 for training, 58 for validation).
  • Cityscapes & BDD100K (Daytime): Used as auxiliary daytime datasets to train the ReLAM light adaptation module and for joint-training experiments.

Key Insights and Ideas

  • Context is King at Night: Standard region proposal networks tightly crop objects. However, at night, a tight crop can strip away the exact context needed for classification. As shown in the paper's comparison between a baseline Region Detection Network (RDN) and the proposed HDM, cropping just a rider makes the model predict "person" because the motorcycle is cut out. HDM explicitly learns to propose regions with their surrounding context, allowing the regional model to correctly identify the "rider" class.
Comparison of region proposal networks showing how HDM captures crucial surrounding context compared to RDN.
  • Texture Preservation over Extreme Enhancement: Simply making a night image look like day via unconstrained GANs introduces artifacts that break segmentation models. Regularizing the light adaptation process (via ReLAM) yields much better generalization.
  • Data Quality Matters: The creation of NightCity+ highlights a persistent issue in niche computer vision tasks: benchmark dataset noise can mask true architectural improvements. Clean validation data was essential to prove NightLab's efficacy on slim, easily missed objects.

Key Results

NightLab achieved state-of-the-art (SoTA) results across multiple configurations:

  • NightCity+ Benchmark: When jointly trained with Cityscapes, NightLab-HDM achieved a 62.82% mIoU, outperforming the strong UPer-Swin baseline (59.67%) and delivering an approximate 10% absolute improvement over the original NightCity baseline.
  • BDD100K-Night Benchmark: NightLab-HDM achieved 50.42% mIoU (with daytime joint training), compared to UPer-Swin's 48.04%.
  • Small Object Recovery: Qualitative results show that NightLab successfully resolves fine details for challenging categories like poles, traffic signs, and riders that concurrent models miss entirely.

Limitations and Future Work

Despite setting a new state-of-the-art, the authors note that nighttime semantic segmentation accuracy still lags significantly behind daytime performance. While NightLab is highly effective for object-like classes (riders, poles, cars), it provides less improvement for background "stuff" classes like terrain, which remain easily confused with vegetation in pitch-black environments. Future work will need to explore even more robust context-mining techniques to completely close the day-night performance gap.

Summary updated

Abstract

The semantic segmentation of nighttime scenes is a challenging problem that is key to impactful applications like self-driving cars. Yet, it has received little attention compared to its daytime counterpart. In this paper, we propose NightLab, a novel nighttime segmentation framework that leverages multiple deep learning models imbued with night-aware features to yield State-of-The-Art (SoTA) performance on multiple night segmentation benchmarks. Notably, NightLab contains models at two levels of granularity, i.e. image and regional, and each level is composed of light adaptation and segmentation modules. Given a nighttime image, the image level model provides an initial segmentation estimate while, in parallel, a hardness detection module identifies regions and their surrounding context that need further analysis. A regional level model focuses on these difficult regions to provide a significantly improved segmentation. All the models in NightLab are trained end-to-end using a set of proposed night-aware losses without handcrafted heuristics. Extensive experiments on the NightCity and BDD100K datasets show NightLab achieves SoTA performance compared to concurrent methods.
BibTeX
@misc{deng2022nightlab,
  author = {Xueqing Deng and Peng Wang and Xiaochen Lian and Shawn Newsam},
  title = {{NightLab: A Dual-level Architecture with Hardness Detection for Segmentation at Night}},
  year = {2022},
  eprint = {2204.05538},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.05538},
}
Figure