MELANIE GONICK

In demonstrations, the team has shown that a person wearing the wristband can wirelessly control a robotic hand. As the person gestures or points, the robot does the same. In a sort of wireless marionette interaction, the wearer can manipulate the robot to play a simple tune on the piano and shoot a mini basketball into a desktop hoop. With the same wristband, a wearer can also manipulate objects on a computer screen—for instance, pinching the fingers together to enlarge and minimize a virtual object.

The researchers are planning to further miniaturize the wristband’s hardware, which is currently similar in size to a cell phone. They also hope to train the AI software on movements from more volunteers with a wider variety of hand sizes, finger shapes, and gestures. 

They envision building a large data set of hand motions that can be plumbed, for instance, to train humanoid robots in delicate tasks such as surgical procedures. The ultrasound band could also be used to let people grasp, manipulate, and interact with objects in design applications, video games, or other virtual settings. Ultimately, the team is building toward a wearable hand tracker that anyone can use to wirelessly manipulate humanoid robots or virtual objects with high dexterity in real time.

“We believe this is the most advanced way to track dexterous hand motion—through wearable imaging of the wrist,” Zhao says. “We think these wearable ultrasound bands can provide intuitive and versatile controls for virtual reality and robotic hands.”