Dynamic Drawing Guidance via Electromagnetic Haptic Feedback

Thomas Langerak, Juan Jose Zarate, Velko Vechev, Daniele Panozzo, Otmar HilligesarXiv 2019 · arXiv

Introduction

Digital drawing has long benefited from post-hoc "beautification" algorithms that smooth shaky lines after they are drawn. However, providing real-time, in-situ guidance remains a significant challenge in human-computer interaction. Existing haptic interfaces for pens often suffer from two major flaws: a loss of user agency and rigid timing constraints. Many systems use permanent magnets that "snap" the pen to a trajectory, making the user feel like a passenger rather than an artist. Furthermore, traditional control systems often force the user to follow a path at a pre-determined speed, "fighting" them if they slow down to add detail.

MagPen addresses these issues by providing "guidance, not governance"—a subtle physical pull that improves accuracy while remaining transparent to the user's creative intent. By employing a novel "time-free" optimization strategy, the system gently pulls the pen toward a desired path while allowing the user to maintain their own drawing speed and style.

The MagPen System

The researchers developed a hardware-software stack called MagPen to explore the potential of dynamic electromagnetic haptics. The hardware consists of a bi-axial linear stage, repurposed from a 3D printer, which moves a high-power electromagnet underneath a Sensel Morph pressure-sensitive tablet. The user interacts with the system using a standard pen equipped with a small permanent neodymium magnet.

The MagPen hardware setup features a 2D linear stage and an electromagnet positioned beneath a digital tablet.

To enable real-time interaction, the system requires a fast and accurate model of electromagnetic forces. Modeling these fields usually requires computationally heavy Finite Element Method (FEM) simulations, which are too slow for interactive use. To overcome this, the authors derived a novel, differentiable analytical dipole-dipole model. This allows the system to calculate the exact force needed at 500Hz, ensuring smooth and responsive feedback.

Control Strategy: Guidance, Not Governance

A core innovation of MagPen is its "time-free" control strategy, implemented via Model Predictive Contouring Control (MPCC). In standard Model Predictive Control (MPC), the system attempts to minimize the distance to a moving setpoint defined by time. In contrast, MPCC treats "progress" along the path as an optimization variable itself.

The system doesn't care when the user reaches a specific point, only that they stay near the intended path. It adjusts the magnet's position and power to "pull" the pen toward the nearest point on the curve. This decoupling of path from time prevents the feeling of being "dragged" and allows the haptic "guide" to essentially wait for the user or speed up to meet them. Furthermore, by using an electromagnet rather than a permanent one, the system can dynamically scale the "stiffness" of the guidance. If the user is far off-path, the force increases; as they get closer, it vanishes, allowing for fine-grained manual control.

Evaluation and Results

The system was evaluated through technical benchmarking and a controlled user study with 12 participants. Users performed various drawing tasks, including primitive shapes like circles and lines, as well as complex figures such as a spiral and a dog's head. These tasks were performed under three conditions: Open-Loop (no guidance), Time-Dependent Closed-Loop (standard MPC), and Time-Free Closed-Loop (the proposed MPCC approach).

MagPen provides real-time haptic feedback across various drawing tasks, significantly improving accuracy.

The results demonstrated significant improvements in drawing performance:

  • Accuracy Gains: Haptic guidance reduced the mean error across complex shapes by approximately 50% compared to unassisted drawing.
  • Precision: The system achieved a mean positional dispersion of just 2.8 mm, proving that electromagnetic guidance can be highly localized and effective.
  • User Preference: In qualitative interviews, participants overwhelmingly preferred the MPCC approach. They reported that standard MPC felt like it was "counteracting" them, while MagPen felt like it provided support while they "still had control."
  • Skill Acquisition: Preliminary observations suggest the system could help with motor skill learning, such as calligraphy or "connect-the-dots" exercises for children, by physically reinforcing correct stroke orders and paths.

Limitations and Future Work

While the current prototype demonstrates the efficacy of electromagnetic guidance, several limitations remain. The hardware speed is currently limited by the stepper motors of the linear stage; very fast, flick-like strokes can outrun the magnet's ability to provide guidance. Additionally, the current desktop form factor is bulky. The authors suggest a future version could use a static grid of electromagnets to eliminate moving parts and make the device tablet-sized.

Future work will also explore incorporating pen tilt sensing. The current implementation assumes the pen is held vertically, but real-world drawing involves varying angles. Finally, integrating predictive modeling to anticipate a user's next stroke could further reduce perceived latency and enhance the feeling of seamless collaboration between the human and the machine.

Summary updated

Abstract

We propose a system to deliver dynamic guidance in drawing, sketching and handwriting tasks via an electromagnet moving underneath a high refresh rate pressure sensitive tablet. The system allows the user to move the pen at their own pace and style and does not take away control. The system continously and iteratively measures the pen motion and adjusts magnet position and power according to the user input in real-time via a receding horizon optimal control formulation. The optimization is based on a novel approximate electromagnet model that is fast enough for use in real-time methods, yet provides very good fit to experimental data. Using a closed-loop time-free approach allows for error-correcting behavior, gently pulling the user back to the desired trajectory rather than pushing or pulling the pen to a continuously advancing setpoint. Our experimental results show that the system can control the pen position with a very low dispersion of 2.8mm (+/-0.8mm). An initial user study indicates that it significantly increases accuracy of users drawing a variety of shapes and that this improvement increases with complexity of the shape.
BibTeX
@misc{langerak2019dynamic,
  author = {Thomas Langerak and Juan Jose Zarate and Velko Vechev and Daniele Panozzo and Otmar Hilliges},
  title = {{Dynamic Drawing Guidance via Electromagnetic Haptic Feedback}},
  year = {2019},
  eprint = {1906.11753},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.11753},
}
Figure