Correction appended October 24, 2012.
When I first met Erin Gee and Mike Winters, their appearances did not fit my preconceived idea of scientific researchers. They were laid back, sipping their coffee; Mike in a Hawaiian chemise and shorts despite the cold weather, and Erin with a feathered earring in one ear and funky eyeglasses. The whole atmosphere they presented spiked my curiosity. They greeted me as I sat down and gathered my things, getting ready to record.
The two made it clear from the beginning that they were not collaborating on a single project, but instead researching the same field of study from two different perspectives. Gee, a researcher at Concordia pursuing her MFA in Studio Arts, is mostly focused on transferring the emotional aspect of voice into robots. Her work is manifested in the form of interactive installations, which are performances that incorporate singing – both her own and that of other people – with robots. She wants to create sounds, filled with emotions, and have them sung by a robot. To do this, Gee uses sensors that take tiny bodily responses to emotions and maps them into numbers – focusing mainly on what happens in the body when an emotion is evoked. “The thing is I’m taking these tiny body performances – the machines can read bodies pretty easily – and we’re putting these numbers into a computer and we’re getting computers to amplify them,” explained Gee. Her overall aim is abstract: to find out what a voice is.
Winters clarified the main difference between their work, stating that “Erin’s more on the artistic side, I’m more on the engineering side.” His work, as a McGill master’s student at the Centre for Interdisciplinary Research in Music, Media, and Technology, is in the field of affective display, which basically conveys human emotion through sound, or “non-speech audio.” His work revolves around taking abstract emotional data and turning it into quantitative data that can be recorded, charted out as a point in emotional “space” and mapped to a certain sound. “That’s my work,” he said, “choosing the sound, basing it on psychological principles.”
I’m sure few people consider mapping emotions into numbers or sounds, but for Gee, that’s the exciting part of the whole area of research, “What I find really fascinating about this field is you take something incredibly personal like an emotion and you see a text file of numbers.”
The stories of how they started out are surprisingly different for two people examining the same unique field. Gee came by it in a stroke of luck, when, after her musical performance (which included robots) at the MARCS Institute, in Sydney – part of the University of Western Sydney – Professor Vaughan Macefield approached her, told her about his research, and asked her to make art with it. She was not the least bit hesitant; how could she reject such an offer? “He [came] up to me and [said he works] in micro-neurography, where he sticks needles into nerves and picks up the electrical fluctuations related to neural activity.” She recalls being bluntly asked, “What can you do with it?” She didn’t know, but still accepted. “Luckily, I had a few months to think about it and we came up with something incredibly simple – just turn the numbers into music.” The robots are the eventual manifestation of this work, relaying emotion through sound.
Winters, on the other hand, a physicist at the start of his research in 2010, was already experimenting with sound as a medium to convey different data. He says his work with emotion began this past summer, after he and his colleague were given a grant to begin working on this project. He worked with a company that already had the technology to detect emotions through their physical manifestations in real time, but they wanted an audio interface for the emotions detected. That’s where Winters came in, with what he hopes will be a long-term project: “I’ll be doing it for my master’s thesis, and also next year if I’m lucky.”
When I asked about the implications of their respective research in the long run, Gee was not surprised. “Most people who have interviewed me are like ‘This is amazing!’ or ‘This is awesome! Show me how I can go to this performance – I really want to go! But what’s the practical use of this?’” Gee hopes there will be practical uses, but explained what she perceived her role to be: “The best example that I can think of…is that in fashion there’s the haute couture runway show where designers are free to express themselves and do the crazy stuff. I consider the scientific application of what I’m developing currently would be [the analog of] what hits the racks, what people would buy, how this affects you in everyday life. But I’m not engaged with that right now, I’m just going to focus on making the first iteration.”
Winters, on the other hand, is “making something that can be sold,” something that concerns human interactions and being able to convey the information provided by emotions to other people as directly and quickly as possible. One of these practical uses is therapy, which he explains rather enthusiastically: “Take people who have autism, for instance – who have emotions but don’t have the facilities to be able to convey those emotions to another person. You, as the parent, or the therapist, would have wearable technology like an earpiece which allows you to listen into a data stream that’s happening inside this person and for you to immediately understand that this person is afraid or disgusted or angry.”
As a non-scientist, I found this research interesting in that it has implications not only for the scientific community, but society as a whole. The two researchers represented two different sides of the same story, combining science and the arts in one field of research. When asked about the coexistence of these two facets in our modern capitalist society, Gee asserted that it is not so much that science is overvalued; it is just that people want a finished product, they want an end, and they want a pill that can make their lives better. To her, science is a form of art, and scientists are artists in their own right. Gee put it best, stating, “It’s really exciting that artists are able to collaborate with scientists, because they have these different priorities, these different historical backgrounds and interests to provide people the things that are inspiring and challenging.” It’s projects like those of Gee and Winters that will start a wave of collaboration between these often-polarized fields.
An earlier version of this article incorrectly listed Gee’s place of work to be the Marx Institute at University of Western Australia, and that Winters’ field of research is in effective display.