Envision this: A jazz band plays on a stage in front of a live audience, and among the performers is a robot. The robot functions as another member of the group, being able to fully improvise jazz with the rest as they play naturally.

Ben Grosser, an assistant professor in the School of Art and Design, is trying to make this scene a reality.

Grosser, along with Kelland Thomas and Clay Morrison at the University of Arizona and Colin Dawson at Oberlin College, is developing a computer system that will be able to listen to a musician play and improvise jazz on its own.

“I’m interested in those areas of human cultural production that we believe are the sole province of humans,” Grosser wrote in an email. “What does it mean when a robot can paint a painting or write a poem or play in a jazz band? Does it simply mean that humans have figured out a way to simulate human activity through computers, or do these systems develop agency for themselves?”

Grosser and his team began working on the project this fall, and he said that the jazz-playing robot should take five years to complete.

He said the computer system will have four parts to it: a corpus of transcribed and encoded jazz solos, analyses of those solos using image schema, a knowledge engine and a real time performance system.

“Our team is interested in jazz improvisation as a model for human-computer communication because it is inherently pre-linguistic,” Grosser explained. “Jazz musicians feel the music and react musically without necessarily stopping to consider what they’ve heard in a language-based sense. It is this reactive and embodied experience of communication through jazz that we see potential in.”

Joan Hickey, lecturer of jazz studies and piano pedagogy at the School of Music, said that playing in a group jazz setting is “one of the only situations where the group is improvising spontaneously in real time without any script or rehearsal.”

“There’s a direction that you’re going, but we don’t know how we’re going to get there and what’s going to happen along the way,” Hickey said. “So there’s this conversation that’s going on between the musicians, and it affects what you play. The ultimate result is this collective product, and it’s very exhilarating.”

She said what is very fulfilling about being able to improvise jazz in that way is that it is “a physical, a social, a mental and an emotional experience on all those levels at the same time.”

Chip McNeill, professor and chair of the jazz studies department in the School of Music, said that “with jazz, the whole point is to be able to do improvisation.”

“The one thing that really distinguishes jazz from a lot of other art forms is the fact that we really want to tap into the combination of the human organic element to it, which is basically the emotional element, the feeling that we have inside ourselves that we would like to express as well as the feeling that we get back from a live audience,” he said.

McNeill said it would be interesting to see what playing with Grosser’s computer system will be like.

“It’s like playing with … a very young musician that has a really limited set of experiences and parameters, at least in the musical realm,” he said. “It wouldn’t be like stepping onto the bandstand with mature musicians or just even young musicians but that have an organic sensibility.”

Hickey said it is hard to predict what it will be like to improvise jazz with the jazz-playing robot because she has not done it yet, but she said she is willing to try it.

“Our intention is that, at the end of the project, a human jazz musician who plays with our system will perceive their interactions as musical communication,” Grosser wrote in an email. “That doesn’t mean it will sound just like a human, but the system should play within the genre commonly understood as jazz. However, instead of emulating well-known human musicians, the system should play with its own style and methods of communication.”

McNeill said that jazz musicians use technologies as tools all the time and that he sees Grosser’s computer system as another tool.

“I don’t see it as another improviser necessarily,” he said. “It may be a tool that improvises and can recreate some aspects of improvisation, but it’s not a complete improviser because it’s impossible to be without it being another human being. Part of improvising means that you need to be human.”

[email protected]