Interpreting sign language with computer vision and deep learning

Interpreting sign language with computer vision and deep learning

Image result for sign language deep learningSign language, the art of communicating with the real world with signs and expressions is majorly used by people with hearing and speaking disability. For a conversation to happen with this language, understanding the signs is as important as conveying the message. Does that mean everyone must know this language to communicate with people with disabilities? Not necessarily.  We have interpreters who can translate what is being said back and forth. Without these interpreters, it’ll be almost impossible to convey their message to us.

Having an interpreter with us always for this purpose is difficult. What if there is a smart interpreter, maybe an app in your smartphone which can do this job more efficiently. With the advancements in the field of computer vision and deep neural networks, this is definitely possible.

The first step is to capture the signs and pre-process them using different computer vision techniques. Next, comes the sign recognition and detection using several deep learning approaches like CNN and RNN. At the end of these steps, we’ll have a translated output of the message. These steps may involve a lot of complications and may not be as simple as they sound.

The above para briefs only half of the task, translating signs to speech or text. The other half where it has to convert text or audio inputs to signs may make more sense in the near future.

The overall steps include:

  1. Capturing the images and pre-processing them.
  2. A neural network to interpret the signs (Convert signs to text).
  3. Taking the message from the user and converting it to sign language.

This article is inspired by Abhishek Singh’s project on ‘Getting Alexa to respond to Sign Language using Webcam and TensorFlow.js’.

https://www.youtube.com/watch?v=_4GybUnRfM8 

https://www.youtube.com/watch?v=kS53y6GWm0w

By: Nagaraj S Murthy

 

Leave a Reply

Your email address will not be published. Required fields are marked *