cross composition, film, dance and AI-assisted media, Cong Liu treats perception not as a fixed fact, but as artistic material capable of moving between the senses.
A colour appears, but rather than being named or described, it is heard. Hue becomes pitch. Brightness alters dynamics. Saturation reshapes timbre. What begins as visual information enters time, acquiring rhythm, density and duration.
This movement from colour into sound formed part of composer and interdisciplinary artist Cong Liu’s doctoral work at the University of Minnesota in 2023. The piece established relationships between hue, brightness and saturation on one side, and pitch, timbre and dynamics on the other, allowing colour to be encountered through hearing rather than sight alone.
The work could be understood as a technical system, but its central question is artistic and personal. Liu is colour-blind. For him, colour has never functioned as the stable, universally shared language that visual culture often assumes it to be. His experience does not simply supply the subject of the work; it unsettles the idea that perception can ever be neutral or identical from one person to another.
Rather than attempting to correct that difference, Liu turns it into a compositional method.
A Different Route into Perception
The translation of colour into sound does not suggest that every colour possesses a scientifically correct musical equivalent. A particular shade does not naturally “belong” to a particular note. The relationships in Liu’s work are constructed, tested and organised by the artist.
The mapping provides a framework, but the experience depends on decisions familiar to composition: register, pacing, contrast, density, duration and the placement of silence. Once visual information becomes sound, it no longer behaves like a static image. It unfolds. It disappears. It can return in altered form. It can remain almost imperceptible or take over the listening space.
This temporal dimension is central to Liu’s practice. Sound does not merely label what is visible; it changes the conditions through which it is experienced.
His interest in movement between artistic forms began long before the doctoral project. Liu grew up in an environment shaped by both music and dance. His mother was a guzheng performer, while his father’s work connected him to movement and performance. Music therefore appeared not as an isolated discipline, but as something already entangled with the body, cultural memory and everyday life.
That early relationship helps explain why Liu’s work has repeatedly crossed between composition, stage performance, film, visual media and technology. The movement between senses in his doctoral project was not a sudden departure from his previous work. It concentrated a question that had been present throughout it: what happens when one artistic language is asked to carry the experience of another?
AI as Material, Not Author
Artificial intelligence entered the project as part of a broader digital and compositional process. Its importance lay not in producing an artwork automatically, but in making relationships between different forms of information available for artistic experimentation.
This distinction matters. Within Liu’s practice, AI is not treated as an independent creative authority. It can process, transform and propose, but it cannot determine why a relationship between colour and sound should exist, how that relationship should develop, or what the resulting experience should mean.
Those decisions remain compositional.
AI-assisted processes can generate numerous possibilities, but abundance is not the same as form. A system may offer several visual or sonic interpretations, yet the artist still has to identify which possibilities possess internal coherence and which are merely novel. Technology expands the field of available material; it does not remove the need to establish necessity.
In this sense, Liu approaches AI much as a composer might approach a new instrument or electronic system. Before it can become artistically useful, its tendencies must be understood. Its limitations must be heard. Its unexpected results must be judged rather than automatically accepted.
The value of the technology lies partly in its instability. A conventional instrument carries an established relationship between gesture and sound. AI systems may produce results that are less predictable, creating an interruption between intention and outcome. That interruption can become productive, but only when it is shaped by an artistic structure.
Liu’s work therefore resists two common descriptions of AI art. It is neither a celebration of effortless machine creativity nor a rejection of technology as inherently opposed to human expression. Instead, AI becomes one material within a larger practice—capable of opening unfamiliar routes through an artistic problem, but never exempt from the artist’s responsibility to choose, organise and revise.
Giving Historical Images a New Sound
This method of translation extends beyond the relationship between colour and hearing.
In 2023, Liu composed and performed live accompaniment for Laborer’s Love, the 1922 silent film directed by Zhang Shichuan, during Il Cinema Ritrovato On Tour–Minneapolis. The film is among the earliest surviving Chinese narrative films, but its presentation to a contemporary American audience required more than the restoration of its images. Live music became part of how the film’s humour, physical movement and historical atmosphere could be experienced in the present.
For the accompaniment, Liu drew on musical material associated with Yue Opera and Shanghai shadow puppetry, as well as the sound of the guzheng. Glissandi, live performance and improvisation interacted with the actions on screen, allowing the score to move between historical reference and immediate response.
The process shares something with the colour-to-sound project. In each case, one medium does not simply decorate another. Sound reorganises the audience’s relationship with what they see.
A silent film is never entirely silent when it is presented with live music. The score affects the perceived speed of a gesture, the weight of a pause and the emotional distance between a century-old image and a contemporary viewer. Liu’s accompaniment does not attempt to disguise that historical distance. It creates a route through it.
The result is a form of cultural translation that avoids reducing Chinese musical language to a decorative sign of identity. The source material operates within the structure and timing of the film, becoming part of its humour and movement rather than appearing as an external cultural label.
Music, Movement and Cultural Memory
A related process appears in Liu’s work for interdisciplinary performance.
In Shanghailanders, original music was integrated with dance, history and public storytelling. The production drew on themes of migration, survival, cultural identity and community memory, with Liu’s composition helping to structure the emotional and cultural movement experienced by the audience.
Here, music again functions as more than accompaniment. It gives time to memory.
Historical information can be communicated through text, images or choreography, but music determines how long an audience remains with a moment. It can allow a gesture to feel unresolved, make repetition accumulate emotional weight or connect separate scenes without explaining their relationship literally.
This is particularly important in work involving migration and cultural memory, where meaning is rarely carried through a single narrative channel. Memory may survive as language, ritual, bodily gesture, melody or fragments of family history. Interdisciplinary performance allows these forms to coexist, but Liu’s approach is not simply to place them beside one another.
The artistic question is what each medium does to the others.
Movement changes when it is placed within a musical pulse. An archival image changes when accompanied by a sound produced in the present. A melody acquires another meaning when connected to migration rather than the context in which it was originally learned.
Translation, in Liu’s work, does not preserve material unchanged. It transforms it.
Beyond the Boundaries of a Medium
The term “interdisciplinary” is often used to describe any project containing more than one artistic form. Liu’s practice suggests a more demanding definition.
Music, visual media, movement and AI are not valuable simply because they appear together. Their combination becomes meaningful when crossing from one medium into another alters the original material.
Colour becomes temporal when translated into sound. Film acquires a live and culturally situated rhythm through musical performance. Historical memory becomes physical through dance. AI-generated possibilities are reorganised through compositional judgement.
Across these projects, Liu’s work repeatedly begins at the point where one medium becomes insufficient. A visual experience may require sound. Historical information may require movement. An emotion may need rhythm before it can be articulated through language.
Technology extends the number of forms available to this process, but the central concern remains artistic rather than technological. The question is not simply what a system can generate. It is whether a transformation between media reveals something that could not be perceived as clearly in the original form.
This also separates Liu’s work from the pursuit of seamless multimedia spectacle. The differences between media are not obstacles to be erased. Sound should not become a perfect substitute for colour, just as music should not dictate a single interpretation of an image.
The tension between them is where the work takes shape.
Perception as an Open Field
Liu’s continuing interest in AI-assisted imagery, interactive sound and immersive environments grows from the same foundation. These technologies make it possible to construct spaces in which sound, image, language and movement respond to one another, and in which the audience may enter the work through more than one sensory route.
Yet the significance of that development will not depend on the number of technologies involved. It will depend on whether those technologies alter how a person listens, watches or becomes aware of their own perception.
The colour-to-sound project is therefore not best understood as an attempt to solve colour-blindness. Nor is it a technical demonstration in which visual data is assigned a musical output.
It is a proposition: that perception is neither fixed nor complete, and that art can create alternative routes into experiences that may otherwise remain inaccessible.
Liu’s practice asks what becomes possible when the boundaries between seeing, hearing and feeling are treated as permeable. Its interest lies not in making every person perceive the world in the same way, but in creating forms through which difference itself can be encountered.
In that space, technology does not replace perception. It reveals how varied perception has always been.
