At the University of Waterloo, researchers are exploring how groups of robots can interpret music and transform it into coordinated visual compositions. Led by Dr. Gennaro Notomista from the Department of Electrical and Computer Engineering, the team has developed a system in which multiple mobile robots respond to musical features in real time, generating dynamic light based artworks on a shared floor canvas.
Raj, T., Sharma, P., Thompson, S., Dien, B. S., & Singh, V. (2026). Green pretreatment strategies for enhanced microbial lipid fermentation and synergistic high-quality lignin recovery for next-generation integrated biorefineries. Chemical Engineering Journal Advances, 25, 101031. https://doi.org/10.1016/j.ceja.2025.101031
The project, described in a recent paper titled Music driven Robot Swarm Painting and presented at the 2025 IEEE International Conference on Advanced Robotics and its Social Impacts, brings together robotics, control theory, and computational analysis of sound. Rather than treating robots solely as tools for industrial tasks, the work positions them as collaborators in creative processes.
Dr. Gennaro Notomista from University of Waterloo stated,
“We included the human control input to allow people and robots to work together. The human painter should complement and be complemented by what the robots do.”
The system consists of several small wheeled robots, each roughly the size of a soccer ball. As they move within a defined area on the floor, they emit colored light trails. A camera positioned above the workspace captures these trails, which accumulate into a visual representation of the music being played. The resulting image can be viewed as a painting shaped not by brushstrokes, but by coordinated robotic motion.
The technical foundation of the project lies in two parallel challenges. The first involves swarm coordination. Controlling multiple autonomous robots within a confined area requires algorithms that prevent collisions, maintain coverage, and allow for collective adaptation. The Waterloo team tested their system with up to twelve robots, though the architecture is designed to scale to larger swarms. Each unit operates under distributed control rules while remaining responsive to shared musical input.
The second challenge concerns translating sound into motion. The system analyzes key musical features such as tempo, rhythm, and chord progression. These elements are processed through signal analysis techniques that extract parameters associated with perceived emotional qualities. The extracted features are then mapped to robotic behaviors, including speed, trajectory curvature, position within the canvas, and properties of the light trails such as color intensity and width.
For example, faster tempos may lead to increased robot velocity or denser motion patterns, while changes in harmony can influence color transitions. The objective is not to produce a literal visualization of waveform data but to create an expressive mapping between auditory input and spatial movement.
An additional layer of interaction allows human participants to influence the painting in progress. Users can adjust certain parameters, such as the thickness of the light trails or the spatial region in which robots operate. According to Notomista, this design choice is intended to promote collaboration rather than replacement. The system frames robots as responsive agents that complement human artistic direction.
Beyond its artistic dimension, the project contributes to research in multi robot coordination. Swarm robotics has applications in environmental monitoring, agriculture, warehouse automation, and search and rescue operations. In these contexts, groups of robots must operate collectively while adapting to dynamic inputs. The music driven painting system offers a controlled testbed for studying synchronization, distributed decision making, and scalability in real time environments.
Previous studies in robotic art have involved single robotic arms or plotters generating drawings from algorithmic inputs. What distinguishes the Waterloo system is the use of decentralized mobile robots working together within shared space. This adds complexity to motion planning and introduces constraints that resemble real world swarm deployment scenarios.
The researchers also note that the project intersects with broader discussions about human robot interaction. As robots increasingly enter public and collaborative spaces, understanding how people perceive and influence their behavior becomes important. The inclusion of adjustable controls in the painting system provides a platform for examining how humans and autonomous systems can co create outcomes.
The paper outlining the system has been made available on the arXiv preprint server, and the team plans to extend the work through user studies and potential public exhibitions. Future experiments may involve professional musicians and visual artists to explore how different genres and performance styles affect the resulting compositions.
While the immediate output of the system is visual art, the underlying engineering questions are rooted in control systems, signal processing, and distributed robotics. By using music as an input signal, the researchers have created a scenario that requires real time interpretation and coordinated response across multiple agents.
In that sense, the project illustrates how experimental platforms outside traditional industrial settings can still advance core engineering principles. Whether applied to artistic installations or field robotics, the capacity to coordinate swarms under dynamic inputs remains a central challenge. The Waterloo team’s work provides one more approach to studying that problem, framed through sound, motion, and light.
Adrian graduated with a Masters Degree (1st Class Honours) in Chemical Engineering from Chester University along with Harris. His master’s research aimed to develop a standardadised clean water oxygenation transfer procedure to test bubble diffusers that are currently used in the wastewater industry commercial market. He has also undergone placments in both US and China primarely focused within the R&D department and is an associate member of the Institute of Chemical Engineers (IChemE).
