Reading emotions

Can computers make the connection?

By Kathryn Balint
STAFF WRITER

June 12, 2001

If you think your computer has a mind of its own, just wait.

Researchers at UC San Diego are creating computer programs that can recognize users' emotions and respond with feelings of their own.

It's the stuff of science fiction, conjuring up images of HAL, the computer in the epic film "2001: A Space Odyssey," which, out of fear of being disconnected, killed human crew members.

Three decades later, in the year in which "Space Odyssey" was set, scientists haven't yet figured out how to make computers more like humans.

Skeptics wonder why they'd even want to, especially given the alarming possibilities raised by HAL.

But sinister machines aren't what scientists have in mind.

Those laying the groundwork for computers with emotions picture a peaceful co-existence.

This much is certain: "It's going to change the way we interact with computers," said Javier Movellan, assistant professor of cognitive science and head of UCSD's Machine Perception Laboratory.

He foresees a future when a computer tutor notices when a pupil is bored or frustrated and then responds with encouragement.

He envisions a day when a robotic pet recognizes when its owner is upset and responds with a sympathetic nudge.

Or the day may come when an automobile senses when a driver is getting too stressed and suggests a break.

"The whole idea is to make the interaction with a machine as close as possible to the way we interact with each other," Movellan said.

Until recently, few people (except maybe science fiction writers) thought about imbuing computers with emotions. The notion seemed absurd.

Then in 1987, computer scientist Rosalind Picard published "Affective Computing," a groundbreaking book that opened up a new school of thought.

Picard suggested that emotions play a critical role in how humans perceive the world and make decisions.

For computers to be truly intelligent, she reasoned, they must be able to comprehend feelings, or perhaps even have their own feelings.

Picard went on to found the Affective Computing Research Group at the Massachusetts Institute of Technology, where she devoted her time to enabling computers to detect and respond to human emotions.

The group's research includes: jewelry and clothing with embedded sensors that detect physical reactions associated with emotion, a computer that responds to its user's frustration and a CD player that plays music based on the listener's mood.

Other computer scientists followed Picard's lead.

IBM's Almaden Research Center, for instance, has created an "emotion mouse" that measures temperature and perspiration levels to determine the user's emotional state.

At UCSD's Machine Perception Laboratory, researchers are developing computers that recognize and react to speech, facial expressions, gestures and body movements.

"If emotions weren't that important," said Movellan, "we wouldn't have them."

For instance, it takes more than a thorough grasp of reading, writing and arithmetic to make a good teacher. A good teacher also shows enthusiasm when a student does well, and displays empathy when a student fails.

"Learning is, in fact, an emotional experience," Movellan said.

Movellan and the researchers who work with him hope to bring that same kind of emotional experience to computer learning.

One of their projects involves

Baldi, an animated computer character that's used to help children with hearing impairments understand speech.

Baldi (who was recently renamed Ms. Gurney) speaks to students, and they speak back.

The students got emotionally attached. Baldi didn't.

After a tutoring session, students made comments like, "Oh, Baldi was sad today" or "Baldi was happy today."

"That's when we realized how important it was to bring emotions into this," Movellan said. "Baldi had no emotions and could not recognize the kids' emotions."

That's where the machine perception lab comes in.

Perception is key

Compared with mapping the human genome or beating the world's best chess player, programming a computer to gauge whether you're happy or sad doesn't sound all that rigorous. After all, a baby, even a dog, knows that much.

Turns out, things that come naturally for us aren't so easy for a pile of silicon and circuits. Things that are easy for computers aren't so easy for us.

Big Blue, the IBM computer that bested world-class chess player Garry Kasparov in a highly publicized match four years ago, may have displayed more brainpower when it came to calculating strategic moves.

But that big hunk of metal still couldn't do something as basic as tell the difference between a pawn and a king.

That requires perception. And programming it into a computer is a Herculean effort.

UCSD researchers look to how the brain works for ideas on how to proceed.

"The power of the brain is vastly more than one of these machines," said UCSD doctoral candidate John Hershey, pointing to one of the desktop computers in the machine perception lab.

A desktop computer, for instance, typically has one microprocessor. By contrast, the brain has about 100 billion microprocessors, in the form of neurons.

Picking out a face

Movellan and his students started with the basics: getting the computer to pick out a human face in a room.

Then they had to get it to follow the face as it moves around a room.

The trick is getting the computer to understand that it's still a face, even if it's turned at an angle.

Then came a lesson in facial features.

UCSD graduate Evan Smith tirelessly marked where the nose, lips and eyes were on thousands of photos so that the computer can "learn" how to identify those parts even on faces it has never seen.

The big challenge? Getting the computer to recognize facial expressions like a smile or a frown -- and to understand what they mean.

The face has 44 muscles and how they're moved and in which combination is very telling about a person's mood. That means the computer has to be taught thousands of possible combinations.

Post-doctoral researcher Marian Bartlett inputs databases of faces showing the full range of human emotion: bored, frustrated, happy, sad, disgusted.

"Our brain recognizes expressions quickly and automatically," Bartlett said. "We don't even think about it."

Hearing voices

The other piece of the puzzle to bringing computers to "life" is programming them to understand human voices and to perceive intonations.

Again, the lab researchers have had to start at the most basic level: getting the computer to differentiate between a voice and background noise.

While voice-recognition software is already on the market, most of it requires a microphone close to the speaker's mouth.

Even then, the very best speech recognition tools miss about 10 percent of the spoken words.

"Imagine if you didn't understand one out of 10 words," said Hershey, the graduate student. He's working on a speech-recognition system that tracks a speakers' lips in an effort to improve the computer's comprehension.

No one claims that machines will one day have the full complement of human emotions.

Movellan says machines will have a "simplified version" of emotions.

And he doesn't necessarily think that all machines will be, or should be, endowed with even that limited emotional capability.

"You don't want your word processor to be emotional," he said.

As for the specter of HAL looming over us, Movellan offers this reassurance: "It won't happen in our lifetimes."