Traditional electrical grids operated on a simple, top-down paradigm: a few massive, centralized power plants pushed electricity down high-voltage transmission lines to passive consumers. Because the generation assets were few and large, grid operators enjoyed near-perfect visibility into the system's state.
Today, the rapid proliferation of behind-the-meter (BTM) distributed energy resources (DERs)—such as residential rooftop solar, home batteries, and smart appliances—has flipped this paradigm on its head. While these technologies are vital for decarbonization, they have created a critical visibility gap at the grid edge.
Traditional Grid: [Central Generation] ───► [Transmission] ───► [Distribution] ───► [Passive Load] (Fully Visible)
Modern Grid: [Central Generation] ───► [Transmission] ───► [Distribution] ───► [ Net Load ] (Blind Spot)
▲ ▼
[BTM Solar] [Batteries]
Because utilities typically only monitor the net load at the substation or smart meter level (the total electricity consumed minus the electricity generated locally), the true underlying demand and generation dynamics remain completely invisible. If a neighborhood consumes 50 kW of power but its rooftop solar arrays generate 45 kW, the substation meter registers a mere 5 kW of demand. If a cloud suddenly covers the neighborhood, that 45 kW of solar generation vanishes instantly. To the grid operator, it looks as though a massive, unpredictable load just turned on, threatening local voltage stability and grid reliability.
To prevent blackouts and manage active distribution networks, researchers are developing methods to peer behind the meter. This work splits into three main fronts: unmasking net load through disaggregation, mapping the hidden physical topology of the distribution grid, and designing control mechanisms that can operate safely in the dark.
Unmasking Net Load via Disaggregation and Remote Sensing
How do you measure generation or appliance usage when you do not have a physical sensor attached to them? The research community approaches this using two distinct lenses: looking from above via remote sensing, and inferring from within via mathematical disaggregation.
Remote Sensing and Physical Inversion
Instead of relying on hardware telemetry from millions of individual solar inverters, we can use a combination of computer vision and physics. The intuition is straightforward: if we know where the solar panels are, what angle they face, and what the weather is, we can simulate their output.
[Aerial Imagery] ──► [Computer Vision] ──► [Panel Capacity & Orientation] ──┐
├──► [Physical Model] ──► Estimated Generation
[Weather Data] ───────────────────────────────────────────────────────────┘
To make this possible, Kasmi et al. developed a massive crowdsourced dataset linking high-resolution aerial imagery with technical metadata like tilt, azimuth, and capacity. By training machine learning models on this data, operators can automatically map unregistered rooftop solar installations from satellite images. Once the physical parameters of the arrays are known, Touron et al. demonstrate that combining these structural maps with real-time weather data and physical conversion models allows operators to estimate unmonitored solar generation with an average error of just 10%, drastically improving situational awareness without installing a single new meter.
Statistical and Machine Learning Disaggregation
Another approach is to take the single, aggregated net load signal and mathematically slice it into its constituent parts. Different energy assets leave distinct statistical signatures: a solar panel's output follows the smooth, predictable arc of the sun (modulated by clouds), while an air conditioner cycles abruptly on and off.
- Solar Disaggregation: Chen et al. leverage these distinct signatures by using a Transformer-based model within a Personalized Federated Learning (PFL) framework. This allows the system to learn regional weather and behavioral patterns to separate solar generation from net load while keeping sensitive consumer consumption data localized and private.
- Feeder-Level Disaggregation: Moving up the grid hierarchy, Ledva et al. prove that an online learning algorithm can disaggregate total substation-level demand to estimate the real-time aggregate power consumption of specific flexible loads, like water heaters or air conditioners, across thousands of homes simultaneously.
- High-Frequency Monitoring: At the household level, Saraswat et al. push this to the extreme by developing a hybrid classification-regression model that operates at 60 Hz (cycle-by-cycle). By identifying the exact moment individual appliances switch on or off, they unlock the ability to utilize residential loads as high-speed flexible assets for grid stabilization services.
Mapping the Blind Infrastructure
The grid visibility gap is not just about what is generating or consuming power; it is also about where that power is flowing. High-voltage transmission lines are meticulously documented, but low-voltage distribution networks are dynamic, frequently reconfigured, and plagued by poor documentation. Without an accurate map of which house is connected to which transformer or phase, injecting massive amounts of solar power can cause localized voltage spikes that damage equipment.
To solve this, researchers treat the distribution grid as a hidden graph that can be inferred by watching how it "vibrates" during normal operation. Every time a house changes its power consumption, it causes a minor voltage fluctuation that ripples through the network. The strength and correlation of these ripples between different nodes depend entirely on the physical impedance and connectivity of the wires connecting them.
[Smart Meter Voltage Fluctuations] ──► [Covariance / PCA Analysis] ──► [Inferred Physical Topology]
By analyzing these ambient dynamics, we can reconstruct the grid's structure without manual inspection:
- Linear and Conservation Approaches: Jayadev et al. apply Principal Component Analysis (PCA) and graph theory to time-series energy measurements from smart meters. By exploiting the principle of energy conservation, their data-driven method maps exactly how consumers are linked to specific phases and network assets.
- Capturing Higher-Order Interactions: Because power flow equations are inherently non-linear, simple linear correlations can miss complex dependencies. Yang et al. overcome this by introducing a self-driven graph Volterra model, which successfully captures higher-order interactions among groups of buses in radial distribution grids.
- Pruning Spurious Connections: In grids that contain loops (loopy topologies), standard statistical models often hallucinate "spurious" connections between nearby nodes that are not actually wired together. Talukdar et al. solve this by combining multivariate Wiener filtering with a phase-response pruning technique to isolate true physical lines from statistical noise.
- Handling Unmonitored Nodes: In real-world distribution networks, many nodes lack any instrumentation. Yuan et al. address this "Inverse Power Flow Problem" by proving that even when certain buses remain completely hidden, mathematical inversion algorithms can still uniquely reconstruct a reduced version of the network's nodal admittance matrix.
Operating and Regulating in the Dark
While disaggregation and topology estimation help close the visibility gap, operators will likely never achieve perfect, real-time omniscience over every asset at the grid edge. Therefore, a major branch of this research focuses on designing control algorithms and market structures that are fundamentally robust to incomplete data.
Feedback Control without Global Visibility
Traditional grid optimization algorithms require real-time data from every single non-controllable load to calculate safe operating limits. Since this is economically and technically unfeasible, recent work focuses on using local feedback loops as a proxy for global information.
Colot et al. introduce a "Feedback-based Safe Gradient Flow" algorithm that regulates renewable energy resources in real time. Instead of requiring a complete map of household loads, the algorithm uses local, real-time voltage measurements to dynamically steer the system away from operational boundaries, guaranteeing safety via Control Barrier Functions. When the network structure itself is completely unknown or changing, Yeh et al. combine nested convex body chasing with robust predictive control. Their framework provides a "finite-mistake" guarantee, ensuring that a distribution grid converges to safe voltage limits even when operating within a completely unmapped, "dark" environment.
Mitigating Market Manipulation
The visibility gap does not just create engineering challenges; it also introduces economic vulnerabilities. In modern electricity markets, third-party "aggregators" bundle thousands of residential solar and battery systems into Virtual Power Plants (VPPs) to sell power back to the grid. However, because the central operator cannot directly observe the local weather or the exact state of charge of these distributed assets, a dangerous verification gap emerges.
[Aggregator] ───(Strategically Curtails Generation)───► [Grid Operator]
│
(Cannot verify if drop was
due to clouds or cheating)
As Ruhi et al. demonstrate, aggregators can exploit this lack of observability by strategically curtailing their generation to artificially inflate local electricity prices. Because a regulator cannot easily verify after the fact whether a sudden drop in solar output was caused by a passing cloud or intentional withholding, traditional market monitoring fails.
To counter this strategic behavior, Robu et al. propose an elegant mechanism design solution using strictly proper scoring rules. By tying an aggregator's financial compensation directly to the accuracy of the probabilistic forecasts they submit, the mechanism makes honest reporting the mathematically optimal strategy. Aggregators are penalized for unexpected deviations unless they accurately predict their own volatility, aligning the financial incentives of private edge-asset owners with the physical reliability needs of the central grid operator.
Key Foundational Work
Cooperative Virtual Power Plant Formation Using Scoring Rules (2012) — contributed 1 concept elements to this direction.
Decentralised Control of Micro-Storage in the Smart Grid (2011) — contributed 1 concept elements to this direction.