Abstract
File reading is the basis for data sharing and scientific computing. However, manual programming for file reading is labour-intensive and time-consuming, as data formats are heterogeneous and complex. To address such an issue, this study proposes a novel approach for the automatic generation of file reading programs based on structured and self-described data format information. This approach provides two modes composed of sequentially and randomly reading. The file data format is described by Data Format Markup Language and thus DFML documents are generated. The formation of data type sequences by parsing those DFML documents. The generation of programs for sequential or random reading data with formed data type sequences and general programing rules for specific programming languages. A tool named DFML Editor was developed for generating and editing DFML documents. Case studies on binary files, i.e., ESRI point shapefiles and plain text files, i.e., input files of Storm Water Management Model, were conducted with the software developed for automatic program generation and file reading. Experimental results show that the proposed approach is effective for automatically generating programs for reading files. The idea in this study is also helpful for automatically writing files.
Introduction: The "Tower of Babel" in Data Formats
In scientific computing and data sharing, researchers often grapple with a "Tower of Babel" of file formats. Whether in geoscience, biology, or chemistry, data is stored in a dizzying array of heterogeneous formats. Traditionally, developers have faced two difficult choices: write a custom, manual parser for every single format—a process that is labor-intensive and error-prone—or rely on "universal" libraries that are often bulky and difficult to maintain when a file format undergoes even minor changes.
The core challenge lies in the tight coupling between the understanding of a data format and the specific syntax of a programming language. This research aims to bridge this gap by creating a machine-readable way to describe data structures that can then be translated into functional source code automatically.
Methodology: The DFML Framework
The researchers proposed a three-step strategy centered around a new XML-based language called DFML (Data Format Markup Language). This framework decouples the structural description of a file from the logic required to read it.
- Structural Description: Users define the file's layout using DFML. This involves identifying data types (integers, doubles, strings), groups (nested structures), separators (for text files), and locations (where the data sits in the file).
- Linearization: The system parses the DFML document to create a "linear data type sequence." This essentially flattens complex, nested file structures into an ordered list of data items.
- Code Generation: Using the linear sequence and a set of predefined programming rules for a target language (C# in this study), the system auto-generates the source code. It handles complex logic like
whileloops for repeating data blocks and byte-order conversions (endianness) for binary files.
To facilitate this process, the authors developed a GUI tool called the DFML Editor, which allows users to visually build and preview these data descriptions without needing to write raw XML.
Key Insights: Linearization and Positioning
The most powerful idea in this approach is treating the file format as metadata. By converting a hierarchical file structure into a flat, linear sequence of data types, the system simplifies the logic required to move a file pointer and extract values.
Furthermore, the system supports two distinct paradigms for data access: sequential and random. This is achieved through different positioning manners within the linear sequence:
- Relative Manner: Used for sequential reading, where each data item's position is relative to the previous one.
- Absolute Manner: Used for random access, where the system calculates the exact byte offset for any specific data item, allowing the program to jump directly to a data point without reading the entire file.
Validation and Results
The authors validated their approach using two distinct, real-world case studies:
- ESRI Point Shapefiles (.shp): A complex binary format used in Geographic Information Systems (GIS) involving headers and variable-length records.
- Storm Water Management Model (SWMM) Input Files (.inp): A structured plain-text format used in hydrology that relies on keywords and separators.
The system successfully generated functional C# code for both formats. In a practical test, the generated code successfully parsed data regarding 13 middle schools in Nanjing, China. These results demonstrate that the approach significantly reduces manual coding effort; instead of writing hundreds of lines of parsing logic, a developer only needs to describe the format once in a DFML document.
Conclusion and Future Work
While the current system effectively automates file reading, the authors acknowledge areas for improvement. The auto-generated code can be somewhat verbose, and future iterations could focus on producing more compact, idiomatic output. Additionally, while the current focus is on C#, the framework's "general programming rules" could be expanded to support languages like Python, C++, or Java. Ultimately, the researchers suggest that this DFML-based logic could be applied to auto-generate file-writing programs, completing the data lifecycle automation.
Summary updated