Smartwatch Gesture Tracking Turns Wearables into Minority Report-Style Controllers

New hand-tracking technology uses sound waves and machine learning to enable air typing, virtual controls, and more on Android smartwatches.

Apr. 10, 2026 at 4:21pm

Got story updates? Submit your updates here. ›

A highly detailed 3D illustration of a smartwatch emitting glowing cyan and magenta sound waves that are bouncing off a human hand, creating a cybernetic, high-tech visualization of the gesture tracking technology at the heart of this story.Smartwatch gesture tracking technology brings Minority Report-style controls to the wrist, transforming wearables into powerful hands-free input devices.Los Angeles Today

Researchers from Cornell University and KAIST have developed a new system called WatchHand that can turn any Android smartwatch with a speaker and microphone into a 3D hand-tracking device, allowing for gesture controls, air typing, and virtual environment navigation without the need for cameras or additional sensors. The technology works by emitting inaudible sound waves that bounce off the user's fingers and palm, with the watch's microphone detecting the echoes and using machine learning to map the 3D position and movements of the hand.

Why it matters

This breakthrough in smartwatch hand-tracking technology has the potential to revolutionize how we interact with our devices, moving beyond traditional touchscreens and keyboards to more natural and intuitive gesture-based controls. If widely adopted, it could enable new applications in areas like virtual reality, smart home controls, and hands-free computing.

The details

The WatchHand system works by using the speaker and microphone already present in many smartwatches to emit and detect sound waves that bounce off the user's hand. An on-device machine learning algorithm then processes these echo patterns in real-time to accurately track the 3D position and movements of the fingers and palm. This allows for a range of gesture-based controls, from air typing to virtual environment manipulation, without the need for cameras or additional sensors.

  • The WatchHand research will be presented at the ACM CHI 2026 conference in Barcelona.
  • The technology is currently limited to Android smartwatches, with iOS support not yet available.

The players

Cornell University

A private Ivy League research university located in Ithaca, New York, known for its strong programs in science, technology, and engineering.

KAIST

The Korea Advanced Institute of Science and Technology, a prestigious public research university in South Korea focused on science and technology.

Chi-Jung Lee

A doctoral student at Cornell University and co-lead author of the WatchHand research.

Jiwan Kim

A doctoral student at KAIST and co-lead author of the WatchHand research.

Got photos? Submit your photos here. ›

What they’re saying

“In the future, with this kind of hand-tracking technology, we might be able to track our typing with just our smartwatch. Our hands can act as an input device with computers.”

— Chi-Jung Lee, Cornell doctoral student and co-lead author

“WatchHand substantially lowers the barriers to hand-pose tracking. If any device has a single speaker and microphone, our approach is applicable.”

— Jiwan Kim, KAIST doctoral student and co-lead author

What’s next

The researchers are continuing to refine the WatchHand technology, including improving its accuracy when the user is walking around and expanding its capabilities beyond the current Android-only limitations.

The takeaway

The WatchHand system represents a significant breakthrough in smartwatch technology, transforming these wearable devices into powerful gesture-control hubs. If widely adopted, it could enable new and more intuitive ways of interacting with our digital devices, from virtual reality experiences to hands-free computing.