Talking in colour

Karrie Karahalios can show a child with Asperger's syndrome when he's lost in a conversational riff or a taciturn spouse when he doesn't speak very much.

Their voice appears on a computer terminal as vibrant colours - red, yellow, blue, green - the image growing in size if the voice gets louder, overlapping another colour as it interrupts or abruptly narrowing with silence.

They are talking in colour.

Karahalios, a computer scientist at the University of Illinois at Urbana-Champaign, has devised a way to digitise conversations and spit them back as images that let people "see" their own conversations on computer monitors.

Social mirror
The Massachusetts Institute of Technology-trained professor says her method provides feedback in real time and can act as a type of social mirror, allowing people to adjust their speech in the same way they adjust their appearance before a glass mirror.

"You look into a mirror and you change your dress, your expression, because you see exactly how it's happening in real time," she said.

The colours linger so people can see the progression of an entire conversation, not only the present moment.

The computer programme, which she calls a "conversation clock", has been tested with low-functioning autistic children and in marriage counselling and is being prepared for use with Asperger's syndrome.

Troubled social interactions
People with that disorder, at the high end of the autism spectrum, often have sophisticated vocabularies but troubled social interactions.

"Kids with Asperger's tend to do 'monologuing' and 'lecturing'" without letting others intervene," said Maria Dixon, a clinical instructor in hearing and speech at the University of Maryland.

"The challenge is to get them visual feedback while this is happening."

The experiment will use the conversation clock programme with two children sitting across from each other at a table.

It will show if children with Asperger's do what other users of the clock tend to do - change their conversational patterns to balance the colours that appear on the computer screen. That would put an end to monologuing, at least during the experiment.

"My older son is fascinated, because he thinks this could really help him," said psycholinguist Sara Weyland, Dixon's colleague and the mother of two children with Asperger's.

Karahalios' team has prepared for a year and will run the study this summer at the College Park, Maryland, campus.

Balanced conversation
Karahalios has been studying the use of computers and visualisation of human speech for more than a decade. She published her work on the "conversation clock" two years ago and is continuing to refine it.

Experts in various fields have suggested a variety of applications, including marriage counselling. She also has a request to use the clock to teach turn-taking to people with a specific kind of brain injury.

Karahalios presented her work this month at the eComm Conference in San Francisco.

She described a test in which four people sat at a table, each fitted with a lapel microphone matched to a colour and attached to a computer. The conversation clock charts a colourful, real-time record of their conversation that is projected on the table, instead of on a wall or a computer screen.

The "clock" shows the progress of the talk. Three times a second, a colour bar pops up showing who was speaking. The louder the speech, the longer the bar. Interruptions are shown as overlapping colour bars. Every minute, a new circle of bars is rendered in a visual record akin to the rings of tree trunk.

Fascinating behaviour
Karahalios found that once it was turned on, people tried to balance their conversation.

"Everyone sees the exact same thing and people almost felt like it had to be a balance of colour," Karahalios said. "I'm not saying that balanced conversation is good conversation. It's fascinating how people behaved."

Once she had established her techniques, Karahalios worked to apply them to children with autism.

Research has shown computers can teach in ways that reduce human-to-human interaction, which tends to be a source of anxiety to people with autism. That came into play in an experiment Karahalios did with some low-functioning autistic children, age 3 and seven, who did not talk.

The idea was to help children "vocalise", using sounds that are the basis of speech instead of screams or grunts.

When a child watching the computer monitor vocalised, a cartoon character would spin, or a cone would erupt in fireworks. The longer the vocalisation, the greater the reaction: the cartoon character spun more quickly, or more fireworks appeared.

Help children and families
As for audio, the computer would give the child an "echo", a short sound for a short sound, a longer one for a longer sound.

"We got him to say 10 words, which his mother had never gotten him to say," said Joshua Hailpern, a doctoral candidate working with Karahalios and two other professors.

Karahalios hopes for the day her work can become widely available to help children and families, through iPhone programmes and as toys for children. A toy for autistic children who have trouble working with doctors, or who are far away from specialised help, could give "rewards" to kids as they play with the toy and make sounds.

"We don't want it to look like a small computer," she said. "It needs to be inviting ... fun and engaging." – (David Lawsky/Reuters Health, April 2009)

Read more:
Asperger syndrome
HIV/Aids Centre