Google’s hand-tracking algorithm could help translate sign language

Share on facebook
Share on google
Share on twitter
Share on linkedin
Time to Read: 3 minutes

Millions of people communicate using sign language, but so far projects to capture its complex gestures and translate them to verbal speech have had limited success. A new advance in real-time hand tracking from Google’s AI labs, however, could be the breakthrough some have been waiting for.

The new technique uses a few clever shortcuts and, of course, the increasing general efficiency of machine learning systems to produce, in real time, a highly accurate map of the hand and all its fingers, using nothing but a smartphone and its camera.

“Whereas current state-of-the-art approaches rely primarily on powerful desktop environments for inference, our method achieves real-time performance on a mobile phone, and even scales to multiple hands,” write Google  researchers Valentin Bazarevsky and Fan Zhang in a blog post. “Robust real-time hand perception is a decidedly challenging computer vision task, as hands often occlude themselves or each other (e.g. finger/palm occlusions and hand shakes) and lack high contrast patterns.”

Not only that, but hand movements are often quick, subtle or both — not necessarily the kind of thing that computers are good at catching in real time. Basically it’s just super hard to do right, and doing it right is hard to do fast. Even with multi-camera, depth-sensing rigs like those used by SignAll have trouble tracking every movement. (But that isn’t stopping them.)

The researchers’ aim in this case, at least partly, was to cut down on the amount of data that the algorithms needed to sift through. Less data means quicker turnaround.

Related Article  Artificial Intelligence discovers antibiotics which can counter drug-resistant diseases

handgesturesFor one thing, they abandoned the idea of having a system detect the position and size of the whole hand. Instead, they only have the system find the palm, which is not only the most distinctive and reliably shaped part of the hand, but is square, to boot, meaning they didn’t have to worry about the system being able to handle tall rectangular images, short ones and so on.

Once the palm is recognized, of course, the fingers sprout out of one end of it and can be analyzed separately. A separate algorithm looks at the image and assigns 21 coordinates to it, roughly coordinating to knuckles and fingertips, including how far away they likely are (it can guess based on the size and angle of the palm, among other things).

To do this finger recognition part, they first had to manually add those 21 points to some 30,000 images of hands in various poses and lighting situations for the machine learning system to ingest and learn from. As usual, artificial intelligence relies on hard human work to get going.

Once the pose of the hand is determined, that pose is compared to a bunch of known gestures, from sign language symbols for letters and numbers to things like “peace” and “metal.”

The result is a hand-tracking algorithm that’s both fast and accurate, and runs on a normal smartphone rather than a tricked-out desktop or the cloud (i.e. someone else’s tricked-out desktop). It all runs within the MediaPipe framework, which multimedia tech people may already know something about.

Related Article  £2.5m trial of AI system to help patients at risk of fatal lung failure

With luck, other researchers will be able to take this and run with it, perhaps improving existing systems that needed beefier hardware to do the kind of hand recognition they needed to recognize gestures. It’s a long way from here to really understanding sign language, though, which uses both hands, facial expressions and other cues to produce a rich mode of communication unlike any other.

This isn’t being used in any Google products yet, so the researchers were able to give away their work for free. The source code is here for anyone to take and build on.

“We hope that providing this hand perception functionality to the wider research and development community will result in an emergence of creative use cases, stimulating new applications and new research avenues,” they write.

Leave a Reply

SUBSCRIBE FOR XTECHALPHA NEWS AND RECEIVE WEEKLY UPDATES

Xtechalpha Xclusive

RSS Latest Technology News

  • MIUI 12 rolls out for Redmi Note 8T global variant October 25, 2020
    The MIUI 12 update is now reportedly rolling out to the global version of Redmi Note 8T. Based on Android 10, the firmware version 12.0.1.0.QCXMIXM is coupled with the October security patches as well. As expected, the latest update brings many new ...
  • M235i Gran Coupe brings Sedan Sass October 25, 2020
    BMW wanted to cover all the bases with their all new 2020 M235i xDrive Gran Coupe. It mixes up all the best parts of a sedan, a crossover and a coupe. Despite its name, the M235i Gran Coupe is a four-door sedan that competes against other European ...
    Jenny White
  • Samsung launches the 8K Festival in India, offers discounts on QLED 8K TVs till October 31 October 25, 2020
    Samsung has launched 'The 8K Festival' under which it is offering its QLED 8K Televisions at discounted prices in India. The South Korean electronics giant will offer these TVs at prices with discounts of up to ₹630,000. Get access to premium Portfolio ...

Follow Us

Google’s hand-tracking algorithm could help translate sign language

by Minakshi Time to Read: 2 min
0
AI Tool to Reshape Treatment by Predicting Cell Behaviors
Get to know the latest updates on exponential technologies, new age industry segments with our weekly XTechalpha Xclusive newsletter straight in your mailbox.