Abstract
Synesthesia effect between music and color is a researching direction which is concerned much. The problem of mismatching between music playing and color had always existed in applications. On the basis of researching music and color, colors which are corresponding to various chords respectively are determined, and the frequency spectrum analysis on chords is performed. The real-time frequency domain transformation to time domain audio signal being played is carried out. The closest chord to audio signal being played can be identified via comparison between a chord spectrum and the spectrum of audio signal being played, and the color corresponding to the audio signal is determined. To improve the accuracy of recognition the method is investigated with which wavelet transform is adopted to de-noise processing of audio signal. Through tests on the audio experiment platform, color mixed effect test, optimization of the audio signal processing parameters, and processing time sequence scheming have been performed. A novel audio player is developed in application to realize the optimal RGB-LED mixed light which automatically presents matching color effect for arbitrary music being played.
Introduction
While "color organs" and music-reactive lights have existed for decades, they often suffer from a lack of true harmonic relevance. Most commercial systems either follow pre-scheduled patterns or flash randomly based on simple volume peaks, creating a "mismatch" where the visual output doesn't reflect the musical structure. This research aims to achieve functional synesthesia—the psychological phenomenon where one sensory stimulus (hearing music) triggers another (seeing color)—by creating a system that identifies the harmonic core of a song in real-time.
The system digitizes 19th-century artistic theories from composers like Rimsky-Korsakov and Alexander Scriabin. For instance, Scriabin’s system associates C major with red and A major with green. As the number of "sharps" in a musical key increases, the associated color shifts from red toward violet, mirroring the increase in frequency in both the audio and light spectrums.
Methodology
Signal Processing Pipeline
The complexity and "instantaneity" of music require sophisticated processing. The core of the system is a reference database of 20 standard musical triads, which cover approximately 90% of modern music. The processing pipeline involves:
- Wavelet Transform: Raw audio signals are non-convergent and noisy. The system employs a wavelet threshold de-noising method to clean the signal before spectral analysis. This step significantly improves recognition accuracy.
- Fast Fourier Transform (FFT): The de-noised signal is converted from the time domain to the frequency domain to identify spectral peaks.
- Pattern Matching: The live audio spectrum is compared against the reference database to find the "closest chord."
Hardware Implementation
The system is built around a custom hardware stack designed for real-time performance. It utilizes an STC15W4K61S4 microprocessor as the central processing unit. Audio playback is handled by a VS1003 audio decoder, while a CH375 interface manages USB and SD card access. The final color output is controlled via three-way Pulse Width Modulation (PWM) to drive RGB-LEDs, allowing for a wide gamut of mixed light effects.
Results and Discussion
The developed prototype successfully demonstrates real-time music-to-color mapping with a sampling interval of 200 to 500 milliseconds, providing a seamless experience for the observer. In experimental tests, specific triads like A major (A-D-F#) were correctly identified and mapped to the corresponding green light effect, validating the spectral comparison algorithm.
By focusing on chord structures (triads) rather than just individual pitches, the system produces a stable and aesthetically pleasing light show that feels "anchored" to the music's harmony. The use of PWM ensures smooth transitions between colors, avoiding the jarring flashes common in simpler reactive systems.
Conclusion
This research provides a novel approach to bridging audio and visual aesthetics through real-time spectral analysis and historical synesthesia mappings. While currently optimized for standard tonalities, future work could expand the reference database to include more complex musical structures found in jazz or experimental genres. The system has broad application prospects in urban landscape lighting, stage production, and psychotherapy, where "color music" can serve as a therapeutic tool.
Summary updated