Support System Using Augmented Reality Display for Use in Teaching the Guitar

Yoichi Motokawa, Hideo SaitôThe Journal of The Institute of Image Information and Television Engineers 2007 · DOI

Abstract

The complex and unfamiliar hand positions required to produce notes on the guitar mean that a beginner player usually finds learning to play a difficult process. We propose a system that makes use of augmented reality. The way to correctly hold the strings is shown by overlaying virtual hand models and lines onto a guitar via a display. By mirroring their hand movements with the image shown on a visual guidance, the player learns the correct hand positions. Accurate registration between the visual guidance and the guitar is a vital part of this system, so we need to track both the hand positions and the position of the guitar. We also propose a method of tracking the guitar by hybridizing a visual marker and the natural features of the guitar. Our tracking method enabled the system to continually display visual guides at the required position.

Introduction

Learning the guitar is notoriously difficult for beginners due to the complex, non-intuitive finger positions required to play chords. While "light-up" guitars exist, they only show where to press, not how to position the hand and fingers. A major hurdle for beginners is the "gaze shift" problem: the constant need to look back and forth between a music book and their hands. This paper introduces an Augmented Reality (AR) system designed to help beginners master the guitar by overlaying virtual 3D hand models and string indicators directly onto a live video feed of the player's instrument.

System Architecture

The proposed system consists of a PC, a webcam, and a display acting as a "virtual mirror." The player sits in front of the display, and the webcam captures the guitar and the player's hands.

The proposed AR system setup featuring a PC, webcam, and display acting as a virtual mirror.

The core technical contribution is a hybrid tracking algorithm designed to solve the "registration" challenge: precisely tracking the guitar's position in real-time to overlay graphics correctly. Standard AR markers are easily obscured by the player's hands (occlusion). The hybrid method combines marker-based tracking for speed and accuracy with natural feature (edge) tracking of the guitar neck when markers are obscured.

Visual Guidance

The system overlays semi-transparent 3D CG hands showing the correct chord fingering. This allows the user to "match" their physical hand to the virtual model without the virtual model completely hiding their own fingers.

Example of the AR overlay showing semi-transparent 3D hand models for chord fingering.

In addition to hand models, colored lines indicate which strings to pluck, and arrows show strumming directions. By using a "virtual mirror" AR display, the player can see instructions and their own hands in the same visual field, significantly reducing cognitive load.

Evaluation and Results

The system was evaluated with 8 beginners learning 9 basic chords and a simple song. Key results included:

  • Improved Speed: Beginners using the AR system were significantly faster, taking an average of 8.9 seconds to switch chords compared to 12.0 seconds with a traditional display.
  • Reduced Eye Movement: Gaze shifts dropped from 3.7 times per chord to just 1.2 times.
  • Tracking Stability: The hybrid method maintained tracking during movements of up to 4.5mm per frame, outperforming marker-only systems during normal play.
  • User Satisfaction: 100% of participants found the AR system more intuitive and effective.

Conclusion

While effective for chords, the 2007 system faced limitations such as a 15 fps frame rate and a lack of detailed support for complex right-hand picking patterns. Future work includes optimizing for single-note melodic lines and moving toward entirely markerless pose estimation.

Summary updated

Abstract

The complex and unfamiliar hand positions required to produce notes on the guitar mean that a beginner player usually finds learning to play a difficult process.We propose a system that makes use of augmented reality.The way to correctly hold the strings is shown by overlaying virtual hand models and lines onto a guitar via a display.By mirroring their hand movements with the image shown on a visual guidance,the player learns the correct hand positions.Accurate registration between the visual guidance and the guitar is a vital part of this system,so we need to track both the hand positions and the position of the guitar.We also propose a method of tracking the guitar by hybridizing a visual marker and the natural features of the guitar.Our tracking method enabled the system to continually display visual guides at the required position.
BibTeX
@article{motokawa2007support,
  author = {Yoichi Motokawa and Hideo Saitô},
  title = {{Support System Using Augmented Reality Display for Use in Teaching the Guitar}},
  journal = {The Journal of The Institute of Image Information and Television Engineers},
  year = {2007},
  doi = {10.3169/itej.61.789},
}
Figure