HYPNOS: Interactive Data Lineage Tracing for Data Transformation Scripts

Xiwen Cai, Xiaodong Ge, Kai Xiong, Shuainan Ye, Di Weng, Ke Xu, Datong Wei, Jiang Long, Yingcai WuIEEE TVCG 2025 · OpenReview

Abstract

In a formal data analysis workflow, data validation is a necessary step that helps data analysts verify the quality of the data and ensure the reliability of the results. Data analysts usually need to validate the result when encountering an unexpected result, such as an abnormal record in a table. In order to understand how a specific record is derived, they would backtrace it in the pipeline step by step via checking the code lines, exposing the intermediate tables, and finding the data records from which it is derived. However, manually reviewing code and backtracing data requires certain expertise, while inspecting the traced records in multiple tables and interpreting their relationships is tedious. In this work, we propose HYPNOS, a visualization system that supports interactive data lineage tracing for data transformation scripts. HYPNOS uses a lineage module for parsing and adapting code to capture both schema-level and instance-level data lineage from data transformation scripts. Then, it provides users with a lineage view for obtaining an overview of the data transformation process and a detail view for tracing instance-level data lineage and inspecting details. HYPNOS reveals different levels of data relationships and helps users with data lineage tracing. We demonstrate the usability and effectiveness of HYPNOS through a use case, interviews of four expert users, and a user study.
BibTeX
@article{cai2025hypnos,
  author = {Xiwen Cai and Xiaodong Ge and Kai Xiong and Shuainan Ye and Di Weng and Ke Xu and Datong Wei and Jiang Long and Yingcai Wu},
  title = {{HYPNOS: Interactive Data Lineage Tracing for Data Transformation Scripts}},
  journal = {IEEE Transactions on Visualization and Computer Graphics},
  year = {2025},
  url = {https://openreview.net/forum?id=1ehEM6u9yN},
}

Loading related content...

Figure