Noncontact and High-Precision Sensing System for Piano Keys Identified Fingerprints of Virtuosity

T. Oku, Shinichi Furuya

TL;DR

This paper introduces a novel, non-contact sensing system for precisely measuring piano key movements at high temporal and spatial resolutions. By analyzing the "fingerprints" of virtuoso pianists' key strokes, they identified specific motor control features that distinguish speed and accuracy, offering new insights into the mechanics of highly skilled piano performance.

What Problem Does This Paper Address?

Understanding the nuances of highly skilled motor actions, like playing a musical instrument, is crucial for fields ranging from neuroscience to robotics. However, precisely measuring the fast, subtle movements involved in such tasks has been a significant challenge. Traditional methods often involve attaching sensors to the tool (in this case, piano keys), which can alter the feel and performance, or using systems with insufficient temporal or spatial resolution. This lack of precise, non-invasive measurement has been a bottleneck in uncovering how exceptional skills like virtuoso piano playing are developed and executed. The paper aims to bridge this gap by providing a system that can capture detailed piano key motion data without interfering with the player's natural touch.

Methods

The core innovation is a custom-made non-contact optical sensing system designed to measure the vertical position of all 88 piano keys. This system uses infrared light projected onto the bottom surface of each key, with photo-reflectors detecting the reflected light. The intensity of the reflected light changes linearly with the distance between the reflector and the key, allowing for precise vertical position tracking.

Key technical aspects include:

  • High Resolution: 1 ms temporal resolution and 0.01 mm spatial resolution.
  • Non-Contact: Sensors are embedded beneath the keys, avoiding any mechanical contact that could alter the playing experience.
  • Data Acquisition: An analog-to-digital converter and a microprocessor collect and process the sensor data, which is then sent to a personal computer via USB.
  • Data Analysis: The collected position data is low-pass filtered. To identify motor skills related to performance, a penalized regression model (elastic net) is employed. This model uses spatiotemporal features of key motions (e.g., peak descending velocity, depression duration) as independent variables to predict performance metrics like maximum speed and loudness accuracy. The elastic net is chosen for its ability to perform variable selection and regularization simultaneously.

Datasets

The study utilized a dataset collected from 49 expert right-handed classical pianists (ages 20-45) with at least 15 years of training. These participants performed a specific experimental task: repeating two sets of simultaneous keystrokes with a one-key interval (a major third), using different finger combinations (right index/ring and right middle/little fingers). This task was performed for 6 seconds at a paced tempo (100 bpm) with a predetermined loudness. The data collected from these performances formed the basis for the analysis. No pre-existing benchmark datasets were used; instead, a new dataset was generated through this controlled experimental setup.

Key Insights and Ideas

  • "Fingerprints" of Virtuosity: The study successfully identified specific, quantifiable "fingerprints" in piano key movements that correlate with a pianist's skill level, particularly in terms of speed and accuracy. This moves beyond simply measuring timing to analyzing the detailed dynamics of key depression and release.
  • Beyond MIDI: The developed system offers significantly higher spatiotemporal resolution than traditional MIDI interfaces, which only capture key press/release timing and a quantized velocity. This allows for the analysis of much finer motor control details.
  • Key Motion Features Matter: The research highlights that not just the overall speed of a keystroke, but also specific aspects like the peak descending velocity, the duration of key depression, and the timing of key release are critical determinants of performance quality.
  • Individual Differences Explained: The penalized regression model effectively explained a significant portion of the variance in performance speed and accuracy based on these identified motor features, suggesting that these are indeed key differentiators among expert pianists.
  • Feedback Control is Crucial: The findings suggest that pianists with better strike-to-strike consistency in their movements (e.g., lower variability in key depression duration) are better at feedback control, likely utilizing somatosensory information from their fingertips.

Key Results

  • The sensing system demonstrated excellent performance with a ground noise ratio of 0.10 ± 0.032% and a linear error of 2.4 ± 0.67%, confirming its accuracy and linearity.
  • The temporal precision was also high, with a standard deviation of sampling time of 8.08 ± 0.02 × 10⁻³ milliseconds.
  • The penalized regression model explained 85.2% (R² = 0.852) of the variance in the Inter-Onset Interval (IOI) at the fastest tempo and 75.9% (R² = 0.759) of the variance in loudness balance.
  • For fastest tempo (IOI), individual differences were primarily associated with the inter-strike mean of peak descending velocity, key depression ratio, and time to key release.
  • For loudness balance (precision), individual differences were linked to the inter-strike variability of peak key descending velocity, key depression ratio, and the inter-strike mean of peak ascending velocity.

Limitations and Future Work

  • Task Specificity: The study focused on a specific chord-trill task. While this task is technically challenging and representative, future work should explore other musical tasks (e.g., scales, arpeggios) to identify broader spatiotemporal features relevant to diverse piano performance.
  • Causality: The current study identifies correlations between motor features and performance. Future research should employ interventional studies to establish causal relationships and understand how to optimize piano practice.
  • Subject Groups: The study focused on healthy young pianists. Expanding to include elderly pianists or those with movement disorders could provide insights into skill deterioration and potential interventions.
  • Technological Refinement: While the system is retrofittable, further exploration of its application to different piano types and potential integration with other sensing modalities could enhance its utility. The authors also suggest investigating other spatiotemporal features like velocity at specific moments, acceleration, and jerk, which might explain tone timbre and contact noises.

Summary updated

Abstract

Dexterous tool use is typically characterized by fast and precise motions performed by multiple fingers. One representative task is piano playing, which involves fast performance of a sequence of complex motions with high spatiotemporal precision. However, for several decades, a lack of contactless sensing technologies that are capable of precision measurement of piano key motions has been a bottleneck for unveiling how such an outstanding skill is cultivated. Here, we developed a novel sensing system that can record the vertical position of all piano keys with a time resolution of 1 ms and a spatial resolution of 0.01 mm in a noncontact manner. Using this system, we recorded the piano key motions while 49 pianists played a complex sequence of tones that required both individuated and coordinated finger movements to be performed as fast and accurately as possible. Penalized regression using various feature variables of the key motions identified distinct characteristics of the key-depressing and key-releasing motions in relation to the speed and accuracy of the performance. For the maximum rate of the keystrokes, individual differences across the pianists were associated with the peak key descending velocity, the key depression duration, and key-lift timing. For the timing error of the keystrokes, the interindividual differences were associated with the peak ascending velocity of the key and the inter-strike variability of both the peak key descending velocity and the key depression duration. These results highlight the importance of dexterous control of the vertical motions of the keys for fast and accurate piano performance.
BibTeX
@inproceedings{oku2022noncontact,
  author = {T. Oku and Shinichi Furuya},
  title = {{Noncontact and High-Precision Sensing System for Piano Keys Identified Fingerprints of Virtuosity}},
  booktitle = {Sensors},
  year = {2022},
  doi = {10.3390/s22134891},
}
Figure