Optimizing Signal Constellation Geometries for Distorted Channels

22 concepts12 papers

In digital communications, data is mapped onto discrete points in a complex signal space known as a constellation. For decades, standard geometric arrangements like Quadrature Amplitude Modulation (QAM) or Phase Shift Keying (PSK) have dominated. These rigid, grid-like structures are mathematically optimal under a simple assumption: that the channel introduces uniform, additive white Gaussian noise (AWGN).

However, real-world channels—ranging from fiber-optic cables to fading wireless environments and low-power backscatter systems—rarely behave so cleanly. They introduce distortions that depend heavily on the signal's amplitude or phase. To maximize capacity and minimize error rates, a growing body of research focuses on optimized and learned constellation geometry. Instead of forcing data into a predefined grid, these techniques dynamically reshape the signal space to match the exact contours of the channel's noise.

The Geometry of Noise: Why Grids Fail

To understand why custom geometries work, consider how different channels distort a signal point. In a coherent optical fiber channel or a high-frequency wireless link, components suffer from nonlinear phase noise (NLPN). This noise doesn't push a signal point equally in all directions; instead, its effect increases with the signal's amplitude, rotating the point along an angular arc. If you use a standard square QAM grid, the outer points (which have higher energy) bleed into each other much faster than the inner points.

As shown by Khoozani et al., the geometric distance between noiseless output constellation points serves as a direct proxy for channel capacity. If we can maximize the minimum distance between these points after they pass through a distorting channel, we can drastically lower the bit error rate and maximize per-user capacity without needing computationally intensive capacity calculations during design.

Structured Optimization: Concentric Rings and APSK

One powerful way to adapt geometry is through Amplitude Phase-Shift Keying (APSK), which organizes signal points into concentric rings. By adjusting three core parameters—the number of rings, the radius of each ring, and the angular spacing of points within each ring—designers can warp the constellation to resist specific channel behaviors.

For instance, in optical channels dominated by NLPN, Häger et al. demonstrated that strategically adjusting ring radii can optimize system performance, even showing that "sacrificing" or omitting certain outer constellation points can prevent phase-induced errors.

Similarly, in low-power backscatter and Reconfigurable Intelligent Surface (RIS) communications, amplitude constraints limit the dynamic range of the transmitter. To combat this, Zhang et al. and Li et al. optimized APSK ring parameters to maximize the minimum Euclidean distance between symbols under strict reflective constraints. By tailoring the constellation to specific channel statistics, the authors significantly boosted reliability in fluctuating fading environments.

Standard 16-QAM (Rigid Grid)       Optimized APSK (Concentric Rings)
       •   •   •   •                             •   •
                                              •         •
       •   •   •   •                        •     • •     •
                                            •     • •     •
       •   •   •   •                        •     • •     •
                                              •         •
       •   •   •   •                             •   •
[Uniform spacing assumes AWGN]     [Rings scale to handle phase noise]

End-to-End Learning via Autoencoders

While structured APSK designs offer a clean mathematical framework, deep learning allows for completely unstructured, data-driven constellation design. By framing a communication system as an end-to-end autoencoder, the transmitter (encoder) and receiver (decoder) can be trained jointly. Here, the coordinates of the constellation points are treated as trainable parameters within the neural network.

This approach allows the system to discover highly non-intuitive, irregular geometric patterns. Letizia et al. showed that capacity-driven autoencoders autonomously learn complex spiral patterns that maintain high reliability in Rayleigh fading channels. Furthermore, by feeding raw channel state information (CSI) directly into a neural network decoder, Luong et al. bypassed traditional mathematical equalizers entirely. This allows the network to simultaneously equalize the channel and discover optimal constellation shapes tailored to specific multi-user or multicarrier fading environments.

This geometric flexibility is also highly effective when paired with Deep Joint Source-Channel Coding (DeepJSCC). As demonstrated by Wang et al., optimizing both the positions of signal points and their decision boundaries simultaneously allows the constellation to natively capture the complex feature representations generated by deep learning models. To bridge these learned shapes with real-world digital hardware, Tung et al. introduced a differentiable quantization mechanism, allowing the network to learn robust, discrete circular constellations that avoid the sudden drop-offs in quality typical of traditional high-order square QAM.

Expanding to Vector Spaces and Multi-User Coordination

The concept of geometric optimization extends beyond two-dimensional complex planes into higher-dimensional vector spaces and multi-user coordination. For example, in multi-user MIMO systems with limited feedback, Jang et al. utilized learned vector quantization to design flexible codebooks in vector space, capturing the underlying statistical properties of the channel far better than rigid binary quantization.

In Over-the-Air Computation (AirComp) for wireless Federated Learning, multiple devices transmit simultaneously so that the receiver decodes the sum of their signals. Standard digital modulation suffers from "summation ambiguity," where different combinations of user data yield identical overlapping signals at the receiver. To solve this, Mu et al. used an autoencoder framework to design custom constellation geometries optimized specifically for over-the-air summation, ensuring that the combined received signal maps uniquely to the correct mathematical sum.

Finally, Li et al. embedded neural mutual information estimators into a cooperative framework to simultaneously calculate a channel's maximum data rate and synthesize the optimal codebook geometry to achieve it, proving more efficient than traditional geometric benchmarks.

The Hardware Trade-off: Structured vs. Unstructured

If irregular, learned constellations consistently outperform standard grids, why aren't they universally adopted? The answer lies in a fundamental tension between theoretical optimality and hardware feasibility.

As noted by Krishnan et al., completely unstructured, irregular signal patterns minimize error rates mathematically but are incredibly difficult to implement in real-world transceivers due to the complexity of calculating arbitrary decoding boundaries. Structured designs, such as concentric APSK rings, serve as a vital middle ground: they constrain points to specific energy levels to simplify transceiver architecture while still providing enough geometric freedom to maximize angular distance and suppress phase-induced errors.

Key Foundational Work

Almost-Optimum Signature Matrices in Binary-Input Synchronous Overloaded CDMA (2010) — contributed 2 concept elements to this direction.
Design of APSK Constellations for Coherent Optical Channels with Nonlinear Phase Noise (2012) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_learned_constellation_geometry_f40adb9e,
  title = {learned constellation geometry},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/learned-constellation-geometry/rdr_f40adb9ee6ed4701ba8e7ec774bc1b68},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_learned_constellation_geometry_f40adb9e_v1,
  title = {learned constellation geometry},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/learned-constellation-geometry/rdr_f40adb9ee6ed4701ba8e7ec774bc1b68/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}

See something wrong? Report a mistake