I-nteract 2.0: A Cyber-Physical System to Design 3D Models using Mixed Reality Technologies and Deep Learning for Additive Manufacturing

Ammar Malik, Hugo Lhachemi, Robert ShortenarXiv 2020 · arXiv

Abstract

I-nteract 2.0 is an innovative cyber-physical system designed to democratize 3D modeling for additive manufacturing. By integrating Mixed Reality (MR), haptic feedback, and deep learning-based reconstruction, the system allows non-experts to create, manipulate, and validate 3D models within their physical environment. This "human-in-the-loop" approach bridges the gap between digital design and physical production, reducing the traditional trial-and-error cycle of 3D printing.

Introduction

Traditional Computer-Aided Design (CAD) software often acts as a barrier to entry for non-experts, requiring specialized training and forcing the design of 3D objects through 2D interfaces. Furthermore, the standard additive manufacturing (AM) workflow is frequently a "trial-and-error" loop where users print a part only to find it doesn't fit its intended physical space. I-nteract 2.0 addresses these challenges by creating an intuitive, immersive workspace where virtual models can be manipulated and validated against real-world constraints in real-time.

The I-nteract 2.0 Framework

The system builds upon its predecessor by offering two primary design pathways within a Mixed Reality environment.

The I-nteract 2.0 hardware setup, featuring a user with a HoloLens and haptic glove interacting with a virtual object.

Constructive Solid Geometry (CSG)

Users can build complex models from scratch using primitive "building blocks" like cubes, spheres, and cylinders. By utilizing a Dexmo haptic glove and VIVE trackers, users can grab, rotate, and resize these shapes using natural hand motions. Boolean operations—Union, Subtract, and Intersect—allow for the merging of these primitives into functional, customized designs.

Deep Learning-Based Reconstruction

For replicating existing objects, I-nteract 2.0 employs a Soft Rasterizer Deep Neural Network (SR-DNN). A user captures a 2D photo of a physical object via HoloLens, which is then processed in the cloud to reconstruct a 3D mesh. To ensure these models are "manifold" (watertight) and suitable for printing, the system uses an Intersection over Union (IoU) metric to match the reconstruction with the closest high-quality model from a pre-existing database.

System Architecture and Integration

The strength of I-nteract 2.0 lies in its end-to-end integration. The system architecture coordinates data flow between the HoloLens, cloud-based AI services, and the physical 3D printer.

System architecture of I-nteract 2.0, showing the data flow between the MR interface, cloud services, and the 3D printer.

Once a design is finalized and "test-fitted" in the virtual space, the system uses OctoPrint to send G-code directly to a 3D printer (such as a PRUSA i3 MK3). This eliminates the need for manual file transfers and external slicing software for the end-user.

Key Insights and Results

The integration of visio-haptic feedback provides a tangible feel to the design process, making it less abstract. More importantly, the "context-aware" nature of the design—where users can resize virtual models to fit physical gaps in real-time—significantly reduces the need for manual measurements and failed test prints.

Validation using the ShapeNet and Pix3D datasets demonstrated that the system can successfully reconstruct 3D models from single-view real-world photos, even with varying backgrounds and lighting.

Limitations and Future Work

While successful, the current SR-DNN is limited to "genus zero" surfaces, struggling with objects containing holes or complex disconnected parts. Additionally, the quality of reconstruction is sensitive to background noise during the initial photo capture. Future developments aim to allow deeper editing of AI-generated models and explore "latent space interpolation" to blend different 3D models into unique new designs.

Summary updated

Abstract

I-nteract is a cyber-physical system that enables real-time interaction with both virtual and real artifacts to design 3D models for additive manufacturing by leveraging on mixed reality technologies. This paper presents novel advances in the development of the interaction platform I-nteract to generate 3D models using both constructive solid geometry and artificial intelligence. The system also enables the user to adjust the dimensions of the 3D models with respect to their physical workspace. The effectiveness of the system is demonstrated by generating 3D models of furniture (e.g., chairs and tables) and fitting them into the physical space in a mixed reality environment.
BibTeX
@misc{malik2020i,
  author = {Ammar Malik and Hugo Lhachemi and Robert Shorten},
  title = {{I-nteract 2.0: A Cyber-Physical System to Design 3D Models using Mixed Reality Technologies and Deep Learning for Additive Manufacturing}},
  year = {2020},
  eprint = {2010.11025},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.11025},
}
Figure