Ways of Seeing / AI Pianist
Ways of Seeing / AI Pianist is an autonomous audio-visual installation that creates a seamless dialogue between artificial intelligence-generated music and responsive digital projections. The work features VirtuosoNet, an AI pianist developed by the Music and Audio Computing Lab at Korea Advanced Institute of Science and Technology (KAIST), performing compositions from contemporary composers (Philip Glass, Olivier Messiaen, Sergei Prokofiev, Dmitri Shostakovich) on a Yamaha Disklavier grand piano. These performances drive a series of immersive visual environments created through the NOS visual engine, which respond in real-time to the musical elements. The installation premiered at the "Ways of Seeing" exhibition at the Daejeon Museum of Art in 2019 and was later showcased at the reopening of Atatürk Cultural Center (AKM) in Istanbul as part of the Beyoglu Culture Road exhibition, supported by the Turkish Ministry of Culture.
Concept
This work explores the relationship between artificial intelligence, musical expression, and visual form through real-time translation. By creating a system where an AI pianist's performance directly influences digital visual environments, the installation becomes what I call an "autonomous art machine" – a self-contained creative system where technology interprets, performs, and visualizes artistic expression without human intervention during the performance.
The installation highlights the intersection between advanced AI research and creative applications. VirtuosoNet, trained on data from 16 composers and 226 performers using a Hierarchical Attention Network (HAN), doesn't simply reproduce scores mechanically but performs with nuanced emotional expression by adjusting beat, rhythm, tone, and touch. This computational interpretation of musical works becomes the foundation for the visual response, creating a unified sensory experience where sound and image exist as complementary expressions of the same artistic impulse.
The physical presence of the self-playing grand piano adds a magical, almost supernatural element to the experience, as audiences witness keys moving and music emerging without a human performer. This creates a unique tension between the physical and digital realms that challenges conventional notions of artistic creation and performance.
Technical Details
At the core of this installation is the NOS visual engine, a custom software system I developed that functions as a visual instrument, analyzing incoming audio and translating different frequency ranges into visual parameters. The system orchestrates the entire installation, sending MIDI signals to the Disklavier piano and controlling the visual presets to ensure perfect synchronization between the AI pianist's performance and the responsive visuals.
The middleware orchestration software manages the timing challenges inherent in such a complex system, ensuring that there's no perceptible lag between the piano's physical key movements, the sound produced, and the visual response. This required careful calibration to account for the different latency characteristics of various piano models used in different exhibition venues.
For each composition, I selected specific visual engines from the NOS library based on the musical character and emotional quality of the piece. For instance, Philip Glass's repetitive patterns were matched with visual environments featuring evolving sequential geometric forms, while more percussive or chaotic compositions triggered particle systems that mimicked natural phenomena like murmurations of birds. Each visual preset establishes the overall aesthetic, while individual elements within the visuals respond dynamically to specific musical parameters – sometimes affecting speed, sometimes color, creating a rich visual counterpoint to the musical performance.
The output of VirtuosoNet is a MIDI file, which the orchestration system feeds to the Disklavier at precisely the right moment, while simultaneously activating the corresponding visual environment. This creates a seamless performance where the AI musician and visual system appear to be communicating directly, though in reality, both are responding to the same underlying computational score interpretation.
Experience
Audiences consistently express amazement at the installation, particularly in witnessing the physical piano performing independently. The sight of keys being pressed without human intervention creates a sense of wonder and makes the invisible processes of artificial intelligence tangible and visceral. The immersive visual environments enhance this experience by making the music visible, allowing viewers to perceive sound and image as a unified sensory field.
The installation creates a space where viewers can contemplate the evolving relationship between technology and artistic expression. Rather than replacing human creativity, the AI pianist and responsive visuals demonstrate how computational systems can become instruments for new forms of aesthetic experience – ones that maintain the emotional depth of traditional arts while exploring possibilities unique to digital media.
Credits
Location: Daejeon Museum of Art, South Korea; Atatürk Cultural Center (AKM), Istanbul, Turkey
Art Direction & Visual Design: Nohlab
Creative Coding & Visual Design: Osman Koç
AI Training & Research: Dr. Jeong Dasaem
Pianist: VirtuosoNet
Real-time Visualization Engine: NOS Visual Engine
Ways of Seeing / AI Pianist represents the convergence of artificial intelligence research and creative technology, demonstrating how computational systems can generate emotionally resonant artistic experiences that bridge the digital and physical realms.
