Overcoming Data Fragmentation and Information Silos

129 concepts132 papers4 proposals

Data is the lifeblood of modern organizations, yet it is routinely trapped. Across domains ranging from industrial manufacturing to public administration, software systems are deployed in isolation. This creates "information islands"—disconnected repositories where valuable data sits underutilized, forming what Jitao Yang et al. describe as "data graveyards."

The challenge of modern data engineering is not merely moving bytes from one database to another, but establishing a shared understanding, trust framework, and access mechanism across these boundaries. To understand why modern integration frameworks work, we must look at how they resolve fragmentation across three distinct layers: semantic, structural, and organizational.

The Root Cause of Information Islands

Why do silos form? It is rarely a technical accident; it is an architectural consequence of organizational boundaries. For instance, in higher education, Kunihiko Takamatsu et al. highlight a persistent disconnect between micro-level classroom insights generated by faculty and meso-level institutional management handled by administrative staff in Beyond Silos: Eduinformatics as a Catalyst for Dissolving Faculty and Staff Boundaries in Higher Education. Because these groups use different tools and maintain separate workflows, their data ecosystems naturally diverge.

Similarly, in industrial settings, real-time operational telemetry (such as SCADA systems) and high-level business logic (such as ERP platforms) run on entirely separate tracks. As noted by Jitao Yang et al. in Pipeline Production Data Model, this operational-business divide creates severe data inconsistencies. When a critical decision must be made, managers are forced to manually reconcile conflicting reports from disparate spreadsheets and legacy systems.

Semantic Interoperability: Building a Shared Language

The most elegant way to integrate data without destroying its original context is to decouple the meaning of the data from its physical storage. Instead of forcing every department to migrate to a single monolithic database, semantic integration layers map the relationships between existing schemas using formal ontologies and metadata registries.

[Legacy System A: SCADA] ----\
                              +--> [Semantic Mapping Layer] --> [Unified Web of Models]
[Legacy System B: ERP] ------/           (Ontologies/RDF)

To achieve this, Lei Zhang 0007 et al. designed the Unified Metadata Registry and Repository (UMRR: Towards an Enterprise-Wide Web of Models). By treating metadata as a "Web of Models" using Semantic Web standards like RDF and Linked Data, the UMRR framework allows organizations to perform cross-tool dependency analysis and global searches without altering the underlying software.

In complex engineering environments where data is scattered across design, manufacturing, and maintenance teams, Falko Schönteich et al. introduced CO-PLM in A Pattern-Based Core Ontology for Product Lifecycle Management based on DUL. This modular ontology provides a foundational vocabulary that bridges the conceptual gaps between different engineering disciplines. By defining formal first-order logic axioms—as demonstrated by Jitao Yang et al.—metadata becomes a first-class citizen capable of automatically reconciling data points from fragmented industrial sources.

Decentralized Trust: Bridging Organizational Boundaries

When information silos exist between entirely different corporations or highly autonomous public agencies, traditional centralized databases fail. No single party wants to cede control or risk data exposure to a central authority. In these scenarios, permissioned blockchains and smart contracts act as an immutable integration fabric.

[Enterprise A] <--- Smart Contract ---> [Enterprise B]
      |                                       |
[Local Ledger] <--- Hyperledger Fabric ---> [Local Ledger]

To facilitate cross-company collaboration without sacrificing data sovereignty, Li Wang et al. developed a decentralized access control framework in Blockchain-based Enterprise Community Data Access Control Methods and Applications. This architecture allows separate enterprises to securely pool data for joint machine learning training or cross-organizational business intelligence while maintaining strict cryptographic ownership over their local records.

This mechanism is equally vital for safety-critical infrastructure and regulated markets:

Unified Infrastructure: Cloud Platforms and Middleware

Where organizational alignment allows for physical or architectural consolidation, researchers focus on infrastructure virtualization. Transitioning from isolated, localized servers to unified cloud platforms or middleware layers abstracts away the underlying hardware complexity.

In higher education, Yang Wenjing et al. demonstrated in Study on College Information Resource Sharing under the Environment of Cloud Computing that moving from isolated departmental servers to a virtualized cloud architecture can drastically boost hardware utilization while breaking down data barriers. To bridge the gap between legacy databases and modern user interfaces without rewriting backend systems, middleware serves as an abstraction layer. Ming Cui utilized a middleware WEB server and JSON-based data exchange in Development Scheme of Mobile Campus Information Integration Platform Based on Android to unify fragmented university systems into a single interface.

Similarly, in public administration, Xuewang Zhang and Wei Shen leveraged Service-Oriented Architecture (SOA) and Single Sign-On (SSO) in Design of Collaborative Platform on Clustering Applications for Governments to allow provincial and local governments to share resources seamlessly, cutting redundant construction costs.

Cyber-Physical Synthesis: Unifying Telemetry and Logic

A particularly challenging form of fragmentation occurs when real-time physical telemetry is disconnected from high-level operational software. Resolving this requires frameworks that ingest heterogeneous data streams into a cohesive analytical model, often manifesting as a digital twin or a centralized data space.

[Physical Sensors] ----\
                        +--> [Unified Data Space] --> [Digital Twin Simulation]
[Workload Logs] -------/

Andrea Borghesi et al. demonstrated this at scale with the M100 ExaData campaign (M100 ExaData: a data collection campaign on the CINECA’s Marconi100 Tier-0 supercomputer), which unified previously siloed supercomputer data—including hardware telemetry, workload logs, and cooling infrastructure—into a single high-resolution dataset for holistic optimization.

In renewable energy, Jinjing An et al. designed an IoT-based traceability platform (An IoT-Based Traceability Platform for Wind Turbines) to link data across a turbine's entire lifecycle, from manufacturing to field maintenance, solving the "information island" problem that hinders fault diagnosis. On a broader scale, Manlio Bacco introduced the METRIQA platform in Agricultural Data Space: the METRIQA Platform and a Case Study in the CODECS project, which uses standardized connectors to aggregate data from IoT sensors and satellites into a decentralized Agricultural Data Space, ensuring data sovereignty for farmers while enabling supply-chain visibility.

Why Integration Works

The reason these diverse approaches succeed is that they reject the naive premise that all data must be copied into a single physical bucket. Instead, they establish interoperability layers—whether semantic (ontologies), cryptographic (blockchain), or architectural (middleware and cloud data spaces). By decoupling data generation from data consumption, these frameworks allow individual departments and systems to retain their operational autonomy while providing a unified, real-time access plane for enterprise-wide intelligence.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_information_silos_918eb24c,
  title = {information silos},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/information-silos/rdr_918eb24cc4704393a9fe7d7012dd9979},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_information_silos_918eb24c_v1,
  title = {information silos},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/information-silos/rdr_918eb24cc4704393a9fe7d7012dd9979/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake