LiLMaps: Learnable Implicit Language Maps

Evgenii Kruzhkov, Sven Behnke

Abstract

Traditional robotic mapping focuses on geometry, leaving robots "blind" to the semantic meaning of their surroundings. LiLMaps introduces a framework for building 3D maps that store language features implicitly within neural network weights. By utilizing a sparse octree structure and an adaptive decoder that learns new objects incrementally, LiLMaps creates compact, searchable maps. Experimental results on the Matterport3D dataset demonstrate that LiLMaps significantly outperforms existing state-of-the-art methods in semantic understanding and robustness to noisy visual-language inputs.

Introduction

For a robot to execute complex natural language commands like "find the blue sofa in the living room," it requires a map that bridges the gap between 3D spatial coordinates and semantic language features. Existing Vision-Language Maps (VLMs) often struggle with three primary challenges: viewpoint inconsistency, where an object is classified differently from different angles; static vocabularies that fail to generalize to novel objects; and high memory requirements for storing dense high-dimensional vectors.

LiLMaps addresses these hurdles by representing the environment through an implicit language map. This approach stores semantic data within the weights of a neural network, allowing for a compact representation that can be updated dynamically as the robot explores new environments.

Methodology

The LiLMaps architecture integrates several key components to achieve efficient and adaptive semantic mapping.

The LiLMaps architecture overview, showing the flow from RGB-D input to the implicit language map via a sparse octree and adaptive decoder.

Sparse Octree Encoding

To maintain a compact memory footprint, LiLMaps utilizes a sparse octree structure. Instead of a dense grid, learnable features are stored only where physical observations exist. The system stores high-dimensional features at the coarsest level and uses low-dimensional corner features for fine-grained spatial detail, balancing resolution with efficiency.

Adaptive Language Decoder Optimization

A core innovation of LiLMaps is its adaptive decoder. Unlike standard approaches that use frozen, pre-trained decoders, LiLMaps optimizes its Multi-Layer Perceptron (MLP) decoder weights on-the-fly when encountering new language features. To prevent "catastrophic forgetting"—where learning a new object class causes the model to forget previous ones—the system employs a replay technique, ensuring a consistent global understanding of the environment.

Measurement Update Strategy

Vision-Language Models (e.g., LSeg or CLIP) are often noisy. LiLMaps implements an exponential smoothing recursive filter to handle these inconsistencies. When an object is re-observed, the system blends the new observation with existing knowledge based on a similarity score, effectively filtering out transient errors or viewpoint-dependent hallucinations.

Experimental Evaluation

The framework was evaluated using the Matterport3D dataset within the Habitat simulator. The researchers compared LiLMaps against established baselines, including OpenScene and VLMaps, using various input types: ground truth labels, simulated semantic segmentation, and real-world features from the LSeg model.

A 3D implicit language map generated by LiLMaps, illustrating the detailed semantic segmentation of a complex indoor environment.

Key Results

  • Semantic Accuracy: In mapping tasks, LiLMaps achieved a Mean Intersection over Union (mIoU) between 71% and 79%, a substantial improvement over OpenScene's 36–41%.
  • Robustness: The system maintained high performance even when processing noisy LSeg features, outperforming 2D-based alternatives like VLMaps.
  • Efficiency: The adaptive optimization process runs at approximately 4 frames per second (fps), with potential for further parallelization to support real-time robotic navigation.

Discussion and Limitations

While LiLMaps represents a significant step forward in semantic mapping, it remains dependent on the quality of the underlying visual-language encoder. If the initial encoder fails to detect an object, the mapping framework cannot recover that information. Additionally, while the system performs exceptionally well in photorealistic simulations, further validation on physical hardware in unstructured real-world environments is necessary. Future work will focus on reducing the computational overhead of the adaptive optimization step to better support high-speed mobile robotics.

Summary updated

BibTeX
@inproceedings{kruzhkov2025lilmaps,
  author = {Evgenii Kruzhkov and Sven Behnke},
  title = {{LiLMaps: Learnable Implicit Language Maps}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00749},
}
Figure