TL;DR
This paper introduces "Meschers," a novel mesh representation that computationally captures the essence of impossible objects (like Escher's art) by focusing on local geometric consistency rather than global 3D embeddability. This allows for standard graphics operations like rendering, relighting, and even intrinsic geometry processing (like distance calculations) to be performed on these paradoxical shapes in a way that aligns with human perception.
What Problem Does This Paper Address?
Impossible objects, famously depicted by M.C. Escher, are visual paradoxes that humans can perceive but which defy a consistent 3D geometric interpretation. Existing computational approaches typically represent these objects by either "cutting" them into separate, locally consistent pieces or "bending" a possible object's geometry to appear impossible from a specific viewpoint. Both methods have significant drawbacks: cutting can disrupt local geometry, hindering operations like smoothing, while bending alters the underlying geometry, making operations like relighting or intrinsic distance calculations produce unintuitive results. The core problem is the lack of a computational representation that captures the perceptual qualities of impossible objects while still supporting standard graphics and geometry processing pipelines.
Methods
The paper introduces "Meschers," a new mesh data structure built upon Discrete Exterior Calculus (DEC). Instead of storing absolute 3D vertex positions, a mescher stores:
- 2D Screen-Space Coordinates: The (x, y) coordinates of vertices as they would appear on a screen, assuming an orthographic camera.
- Relative Depth Offsets ($\zeta$): For each edge, a value representing the directed difference in depth between its two connected vertices. This value is constrained to integrate to zero around individual triangles (local integrability), but not necessarily around larger loops, which is the key to representing impossibility.
This representation allows for the application of DEC machinery. A mescher's depth information ($\zeta$) can be decomposed into a curl-free component (representing possible objects) and a harmonic component (representing the "impossible" aspect). The paper also introduces a depth ordering graph to handle globally disconnected impossible structures.
Key operations enabled by this representation include:
- Rendering and Relighting: Compatible with directional and environment map lighting.
- Laplacian Smoothing: Applied to both 2D coordinates and depth values.
- Heat Diffusion and Geodesic Distance Queries: Leveraging the discrete metric and DEC operators.
- Inverse Rendering: Optimizing a mescher's geometry to match a target image.
- Conversion to Cut/Bent Representations: Demonstrating how meschers generalize existing methods.
Datasets
Not discussed in the paper. The paper focuses on creating and processing novel impossible object geometries rather than evaluating on existing datasets.
Key Insights and Ideas
- Perception is Local: The core insight is that human perception of impossible objects relies on locally consistent cues, not a globally consistent 3D embedding. Meschers directly model this by relaxing global embeddability.
- DEC as a Natural Language: Discrete Exterior Calculus provides a powerful and elegant framework for defining and manipulating differential quantities on meshes, making it ideal for representing and processing the nuanced geometry of meschers.
- Harmonic Component as the "Impossible" Factor: The decomposition of the depth field into curl-free and harmonic components clearly separates the "possible" geometric aspects from the "impossible" ones, offering a principled way to understand and manipulate these paradoxes.
- Unified Representation: Meschers act as a unifying framework, capable of representing both possible and impossible objects, and can be converted into previously used "cut" and "bent" representations, demonstrating their generality.
Key Results
The paper demonstrates the effectiveness of Meschers through various examples and operations:
- Rendering and Relighting: Figures 1, 3, 5, and 8 showcase visually compelling renderings of impossible objects under different lighting conditions, highlighting the perceptually accurate relighting.
- Intrinsic Geometry Processing: Figure 1 and 3 show successful computation of geodesic distances on impossible objects, a feat not easily achievable with prior methods. Figure 6 demonstrates Laplacian smoothing applied independently to screen-space coordinates and depth, or jointly.
- Inverse Rendering: Figure 7 presents a proof-of-concept where a mescher is optimized to match a target image of an impossible object, verifying the recovery of a non-zero harmonic component.
- Qualitative Comparison: Figure 8 provides a qualitative comparison against "cut" and "bent" representations, showing that meschers avoid artifacts in relighting, smoothing, and distance calculations where the other methods fail.
Limitations and Future Work
- Orientability: The current DEC formulation requires orientable and manifold meshes. Representing non-orientable impossible objects (like the "IMPOSSIBLE LETTUCE" mentioned) would require alternative formulations.
- Modeling Interface: Meschers are currently created by converting existing 3D models or via inverse rendering. Developing an intuitive user interface for directly modeling meschers remains an open challenge.
- Inverse Rendering Improvements: While a proof-of-concept, the inverse rendering pipeline could be further improved, potentially by integrating user painting of harmonic components.
- Rendering Enhancements: Experimentation with rotating meschers and incorporating shadows or transparency could enhance rendering capabilities.
- Extrinsic Operations: Exploring other extrinsic geometry processing operations beyond those presented.
- Higher-Order DEC: Investigating the use of higher-order DEC for more complex operations.
- Implicit Representations: Exploring the possibility of an implicit mescher-like representation.
- Vision Science Applications: Further research into how mescher-computed shortest paths align with human perception and their use in understanding visual processing.
Summary updated