Smartwatches Gain Hand-Tracking Capabilities

Cornell and KAIST researchers develop AI-powered micro sonar to enable gesture control on off-the-shelf wearables.

Apr. 7, 2026 at 2:43am

Got story updates? Submit your updates here. ›

A highly detailed, glowing 3D illustration of a smartwatch emitting neon-colored sound waves that interact with a human hand, conceptually representing the micro sonar technology that powers the WatchHand hand-tracking system.Smartwatch sonar technology enables new possibilities for natural hand-based interactions.Ithaca Today

Researchers at Cornell University and the Korea Advanced Institute of Science and Technology have created a new wearable technology called WatchHand that equips standard smartwatches with AI-powered micro sonar capable of tracking hand movements in real-time. This breakthrough could enable new assistive technologies for users with limited mobility or speech, as well as serve as a controller for augmented and virtual reality environments.

Why it matters

Existing wearable hand-tracking prototypes have required bulky additional hardware, making them impractical for everyday use. WatchHand's ability to leverage the existing microphone and speaker within standard smartwatches represents a major step forward in making hand-tracking technology more accessible and ubiquitous.

The details

WatchHand uses the smartwatch's speaker to emit inaudible sound waves that bounce off the user's hand and back into the microphone, creating an echo profile that the system's machine learning algorithm can interpret to estimate the 3D hand pose in real-time. All processing is done locally on the watch, ensuring personal data is not shared.

  • The WatchHand system was tested over 36 hours of gesture data with 40 participants across four studies.
  • The research paper on WatchHand will be presented at the ACM CHI conference on Human Factors in Computing Systems in 2026.

The players

Chi-Jung Lee

A doctoral student in the field of information science at Cornell University and co-lead author of the WatchHand research paper.

Cornell University

A private research university located in Ithaca, New York, where the WatchHand technology was developed.

Korea Advanced Institute of Science and Technology (KAIST)

A public research university in South Korea that collaborated with Cornell on the development of the WatchHand system.

Got photos? Submit your photos here. ›

What they’re saying

“In the future, with this kind of hand-tracking technology, we might be able to track our typing with just our smartwatch.”

— Chi-Jung Lee, Doctoral student, Cornell University

What’s next

The researchers noted that while WatchHand performed well in their tests, the current prototype only works on Android smartwatches and had trouble registering hand poses if the user was walking. Future development will focus on improving the system's robustness and expanding compatibility to other wearable platforms.

The takeaway

WatchHand represents a significant breakthrough in making hand-tracking technology more accessible and practical for everyday use. By leveraging the existing hardware in standard smartwatches, the researchers have paved the way for new assistive applications and immersive experiences controlled by natural hand gestures.