Children with autism often have trouble identifying emotional states of people around them, which makes it hard for them to distinguish the difference between a happy face and angry face. To help this process, some therapists use a friendly robot to help demonstrate emotion in a more engaging way. The child then practices imitating different emotions and responding to each emotion in a fitting way.
This type of therapy is best used when the robot can interpret the child’s behavior and decipher if the child is interested and attentive during the session. Now, researchers at the MIT Media Lab have developed a personalized machine learning tool that helps therapy robots measure a child’s engagement and interest, using data that is unique to each child.
“The long-term goal is not to create robots that will replace human therapists, but to augment them with key information that the therapists can use to personalize the therapy content and also make more engaging and naturalistic interactions between the robots and children with autism,” Oggi Rudovic, a postdoc at the Media Lab said.
One famous saying says, “If you have met one person, with autism, you have met one person with autism.”
This is important in understanding how individualized therapy is vital for these children and how AI can help guide better engagement.
“The challenge of creating machine learning and AI that works in autism is particularly vexing, because the usual AI methods require a lot of data that are similar for each category that is learned. In autism where heterogeneity reigns, the normal AI approaches fail,” Rosalind Picard, co-author on the paper said.
Robot-assisted therapy for autism starts with a human therapist showing a child flash cards of different expressions and teaching them to recognize different emotions. Then, the therapist uses the robot to show these same emotions and the robot observes the child’s reactions.
The researchers used SoftBank Robotics NAO humanoid robots that were two-feet tall and resembled an armored superhero. By changing its eye color, the motion of its body and tone of voice, the robot portrays different emotions.
The study consisted of 35 children with autism, 17 from Japan and 18 from Serbia, and ranged from 3 to 13 years of age. Their reactions to the robot ranged from boredom and sleepiness to jumping in excitement and clapping their hands. The majority of children reacted in a respectful manner to the robot and related to it as if it were a real person, especially during storytelling when the therapist asked how NAO would feel if it were given an ice cream treat.
“Therapists say that engaging the child for even a few seconds can be a big challenge for them, and robots attract the attention of the child,” said Rudovic. “Also, humans change their expressions in many different ways, but the robots always do it in the same way, and this is less frustrating for the child because the child learns in a very structured way how the expressions will be shown.”
The MIT research team realized that deep learning would be useful for therapy robots in order to analyze the child’s behavior in a more natural assessment.
“In the case of facial expressions, for instance, what parts of the face are the most important for estimation of engagement?” Rudovic says. “Deep learning allows the robot to directly extract the most important information from that data without the need for humans to manually craft those features.”
Rudovic and his team built a personalized framework system that collected data on each individualized child. The researchers can then capture video of the child’s facial expressions, body movements, gestures, and collect data on heart rate, body temperature and skin sweat response from a wrist monitor.
The networks and data accumulated improved the robot’s automatic estimation of the child’s behavior for the majority of the children in the study. This allowed for individual feedback on each child rather than a “one-size-fits-all” approach.
Researchers were also able to understand how the deep learning network made estimations, uncovering some cultural differences between the children.
“For instance, children from Japan showed more body movements during episodes of high engagement, while in Serbs large body movements were associated with disengagement episodes,” Rudovic said.
Overall, they hope these developments provide a more individualistic approach to therapy in children with autism.