The Liver Tumor Segmentation Benchmark (LiTS) provides a large-scale, multi-center dataset to standardize the evaluation of automated liver and tumor segmentation in CT scans. While algorithms have nearly mastered segmenting the liver itself (Dice score 0.963), accurately delineating and detecting small, low-contrast tumors remains a significant challenge for the medical AI community.
What Problem Does This Paper Address?
Liver cancer is a leading cause of cancer-related mortality worldwide. For clinicians, accurately measuring tumor size and "tumor burden" (the ratio of tumor to healthy liver tissue) is critical for diagnosis, surgical planning, and monitoring treatment response via protocols like RECIST.
However, manual segmentation is a bottleneck: it is time-consuming, tedious, and suffers from high inter-observer variability. Automating this via computer vision is notoriously difficult because:
Low Contrast: Tumors often have similar Hounsfield Unit (HU) values to the surrounding liver tissue (hypo- or hyper-dense lesions).
Structural Variability: Tumors vary wildly in size, shape, and number (from a single large mass to dozens of tiny nodules).
Background Noise: Chronic liver disease can change the organ's texture, making it harder to distinguish from actual lesions.
The LiTS benchmark was created to provide a common ground for researchers to develop and compare robust, fully automated solutions.
Methods
The paper summarizes the results of 75 algorithms submitted across three major challenge events (ISBI 2017, MICCAI 2017, and MICCAI 2018).
Architectural Shift: The benchmark captured a pivotal moment in medical imaging. Early submissions used traditional machine learning or Statistical Shape Models (SSMs), but by 2018, Deep Learning—specifically variants of the U-Net—became the undisputed standard.
Dimensionality: Researchers experimented with 2D, 3D, and "2.5D" approaches. 2.5D models provide a stack of adjacent slices as a multi-channel input to a 2D network, attempting to capture 3D context without the massive computational overhead of full 3D convolutions.
Refinement: Many top-performing methods used a "coarse-to-fine" strategy: one network first isolates the liver, and a second network focuses exclusively on finding tumors within that region. Post-processing techniques like Conditional Random Fields (CRFs) and morphological filtering were also common to clean up noisy predictions.
Datasets
The LiTS dataset consists of 201 abdominal CT volumes collected from seven clinical sites worldwide (including institutions in Germany, the Netherlands, Canada, Israel, and France).
Training/Test Split: 131 volumes were released for training (with expert annotations), and 70 were held back for "blinded" testing.
Diversity: The set includes primary tumors (like HCC) and secondary metastases (from breast or lung cancer).
Comparison: Compared to previous benchmarks like SLIVER’07 or LTSC’08, LiTS is significantly larger and offers more complex, real-world clinical cases.
Key Insights and Ideas
The "Small Lesion" Struggle: The most striking "aha moment" was the performance gap based on lesion size. While algorithms did well on large tumors, the F1 score for detecting small lesions (under 10mm) was a dismal ~0.10.
Segmentation $\neq$ Detection: A high Dice score (which measures volumetric overlap) doesn't necessarily mean an algorithm is good at finding all tumors. Some models achieved high overlap by segmenting one large tumor perfectly while missing five smaller ones entirely.
Contrast is King: Performance dropped sharply when the contrast difference between the tumor and liver was below 20 HU. This highlights a need for better pre-processing or networks that can "see" subtle texture differences.
3D Wins Out: As hardware improved between 2017 and 2018, full 3D deep learning models (like nn-Unet) began to consistently outperform 2D and 2.5D counterparts by better capturing the volumetric nature of human anatomy.
Key Results
Liver Segmentation: Nearly solved, with the best Dice score reaching 0.963.
Tumor Segmentation: Much harder, with top Dice scores improving from 0.674 (ISBI 2017) to 0.739 (MICCAI 2018).
Tumor Detection: The best lesion-wise recall reached 0.554 in 2018, meaning even the best models missed nearly half of the individual tumor nodules.
Inter-rater Agreement: The study found a median Dice of 70.2% between different human experts for tumor segmentation, suggesting that the "ground truth" itself is often ambiguous.
Limitations and Future Work
The authors acknowledge that the dataset was annotated by only one rater per center, which may introduce label bias. Future iterations should include multiple expert annotations to help models learn to handle clinical uncertainty.
Additionally, the dataset lacks detailed metadata (like scanner types or patient demographics) that could help analyze why certain models fail on specific hardware. The authors suggest that Federated Learning and Domain Adaptation are the next frontiers, allowing models to learn from diverse hospital data without compromising patient privacy.
Summary updated
Abstract
In this work, we report the set-up and results of the Liver Tumor Segmentation Benchmark (LiTS), which was organized in conjunction with the IEEE International Symposium on Biomedical Imaging (ISBI) 2017 and the International Conferences on Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2017 and 2018. The image dataset is diverse and contains primary and secondary tumors with varied sizes and appearances with various lesion-to-background levels (hyper-/hypo-dense), created in collaboration with seven hospitals and research institutions. Seventy-five submitted liver and liver tumor segmentation algorithms were trained on a set of 131 computed tomography (CT) volumes and were tested on 70 unseen test images acquired from different patients. We found that not a single algorithm performed best for both liver and liver tumors in the three events. The best liver segmentation algorithm achieved a Dice score of 0.963, whereas, for tumor segmentation, the best algorithms achieved Dices scores of 0.674 (ISBI 2017), 0.702 (MICCAI 2017), and 0.739 (MICCAI 2018). Retrospectively, we performed additional analysis on liver tumor detection and revealed that not all top-performing segmentation algorithms worked well for tumor detection. The best liver tumor detection method achieved a lesion-wise recall of 0.458 (ISBI 2017), 0.515 (MICCAI 2017), and 0.554 (MICCAI 2018), indicating the need for further research. LiTS remains an active benchmark and resource for research, e.g., contributing the liver-related segmentation tasks in \url{http://medicaldecathlon.com/}. In addition, both data and online evaluation are accessible via \url{www.lits-challenge.com}.
BibTeX
@article{bilic2019liver,
author = {Patrick Bilic and Patrick Christ and Hongwei Bran Li and Eugene Vorontsov and Avi Ben-Cohen and Georgios Kaissis and Adi Szeskin and Colin Jacobs and Gabriel Efrain Humpire Mamani and Gabriel Chartrand and Fabian Lohöfer and Julian Walter Holch and Wieland Sommer and Felix Hofmann and Alexandre Hostettler and Naama Lev-Cohain and Michal Drozdzal and Michal Marianne Amitai and Refael Vivantik and Jacob Sosna and Ivan Ezhov and Anjany Sekuboyina and Fernando Navarro and Florian Kofler and Johannes C. Paetzold and Suprosanna Shit and Xiaobin Hu and Jana Lipková and Markus Rempfler and Marie Piraud and Jan Kirschke and Benedikt Wiestler and Zhiheng Zhang and Christian Hülsemeyer and Marcel Beetz and Florian Ettlinger and Michela Antonelli and Woong Bae and Míriam Bellver and Lei Bi and Hao Chen and Grzegorz Chlebus and Erik B. Dam and Qi Dou and Chi-Wing Fu and Bogdan Georgescu and Xavier Giró-i-Nieto and Felix Gruen and Xu Han and Pheng-Ann Heng and Jürgen Hesser and Jan Hendrik Moltz and Christian Igel and Fabian Isensee and Paul Jäger and Fucang Jia and Krishna Chaitanya Kaluva and Mahendra Khened and Ildoo Kim and Jae-Hun Kim and Sungwoong Kim and Simon Kohl and Tomasz Konopczynski and Avinash Kori and Ganapathy Krishnamurthi and Fan Li and Hongchao Li and Junbo Li and Xiaomeng Li and John Lowengrub and Jun Ma and Klaus Maier-Hein and Kevis-Kokitsi Maninis and Hans Meine and Dorit Merhof and Akshay Pai and Mathias Perslev and Jens Petersen and Jordi Pont-Tuset and Jin Qi and Xiaojuan Qi and Oliver Rippel and Karsten Roth and Ignacio Sarasua and Andrea Schenk and Zengming Shen and Jordi Torres and Christian Wachinger and Chunliang Wang and Leon Weninger and Jianrong Wu and Daguang Xu and Xiaoping Yang and Simon Chun-Ho Yu and Yading Yuan and Miao Yu and Liping Zhang and Jorge Cardoso and Spyridon Bakas and Rickmer Braren and Volker Heinemann and Christopher Pal and An Tang and Samuel Kadoury and Luc Soler and Bram van Ginneken and Hayit Greenspan and Leo Joskowicz and Bjoern Menze},
title = {{The Liver Tumor Segmentation Benchmark (LiTS)}},
journal = {Medical Image Analysis},
year = {2019},
doi = {10.1016/j.media.2022.102680},
}