Hand gesture recognition

 Introduction: 

Computer vision is an area that seeks to facilitate how a computer understands images or videos, through this, it is intended that they focus on performing tasks that humans perform such as object detection, automatic inspection, car navigation, etc. For the present practice we focus only on one area, the motion detection and the subtraction of the background of a video in real time, this without making use of the high-level function that openCV provides.

Inclusive education is one of the transcendental government issues that seeks to achieve an egalitarian education, however progress is not yet considerable, even more so for those students with a disability who are found in the few special basic education centers in the country, based on research focuses on schoolchildren with hearing disabilities, who have sign language their only means of communication and interaction with others, however, this language is not very widespread in our society.

1. Description Of The Problem:

             1.1 Hearing Impairment

Access to sign language, in areas such as education and services public, it is fundamental to the human rights of deaf people, according to the page of the National Council for the Equality of Disabilities (CONADIS), in Ecuador there are 63,191 people with hearing disabilities registered. 54.49% are men and 45.51% are women(1).
Based on this problem, an example is taken as a research work that proposes the design of a hand gesture recognition system based on computer vision for schoolchildren with hearing disabilities, which will serve as a technological instrument of communication between the teacher. and his deaf student, which allows him to participate in a class and learn at the same level as others.

 

            2. Concepts

2.1 What is C++?

C++ is a cross-platform language that can be used to create high-performance applications.

C++ was developed by Bjarne Stroustrup, as an extension to the C language.

C++ gives programmers a high level of control over system resources and memory.

The language was updated 3 major times in 2011, 2014, and 2017 to C++11, C++14, and C++17.

 

2.2  Why Use C++

C++ is one of the world's most popular programming languages.

C++ can be found in today's operating systems, Graphical User Interfaces, and embedded systems.

C++ is an object-oriented programming language which gives a clear structure to programs and allows code to be reused, lowering development costs.

C++ is portable and can be used to develop applications that can be adapted to multiple platforms.

C++ is fun and easy to learn!

As C++ is close to C# and Java, it makes it easy for programmers to switch to C++ or vice versa

 

2.3  OpenCV

OpenCV is an open-source C++ library for image processing and computer vision, originally developed by Intel, later supported by Willow Garage and is now maintained by Itseez. It is free for both commercial and non-commercial use. Therefore, you can use the OpenCV library even for your commercial applications. It is a library mainly aimed at real time processing. Now it has several hundreds of inbuilt functions which implement image processing and computer vision algorithms which make developing advanced computer vision applications easy and efficient. 

                    

                    2.4 Visual Code

Learning to code is intimidating, so set yourself up for success with a tool built for you. Visual Studio Code is a free coding editor that helps you start coding quickly. Use it to code in any programming language, without switching editors. Visual Studio Code has support for many languages, including Python, Java, C++, JavaScript, and more. Ready to get started? Check out these introductory videos or check out our coding packs for Java, Python, and .NET.

                    2.5 Stereoscopic Vision

                         

Stereoscopic vision can be understood as the ability to infer information of a 3D scene from two images taken from different points of view. This is achieved by placing two cameras side by side to obtain two different views of the same scene. This process is similar to the process performed by binocular vision in people. (3)

The principle is easily observed if the extended hand is held with the thumb raised, then one should stare at the finger with alternating sight, that is, first with one eye and then with the other. In each view the finger appears off a certain amount, this amount is called disparity. Disparity is what creates the sense of depth and is the same principle used for stereoscopic vision.

Epipolar Geometry is used in stereoscopic vision to help solve the correspondence problem by limiting the search space of the points by geometrically relating the points of an image to the points of the stereo image.

Two-dimensional search becomes a one-dimensional search as it ensures that the correspondence of a point with vertical coordinate yl in the left image is found in the coordinate row yr = yl of the right image called "scan line” 

                    2.6 Neural Networks

Neural networks correspond to an approach to artificial intelligence that aims to model the human brain. Within the brain are neurons, which are processing units that operate in parallel. There are an estimated ten trillion neurons in the human brain and about sixty trillion connections between neurons. Each neuron receives as input small electrical signals from other neurons and in return also responds with electrical signals. These responses are evaluated in the sense that a neuron does not "fire" an output unless a certain threshold called bias has been reached. The evaluation measure can vary through experience. The brain is therefore a network of neurons acting in parallel.

 

                    2.7 Library 

• Include <ssstream>: Stringstreams. They're like streams, but they contain strings, which can be useful for employing stream-style manipulation on them. Hence, stringstream. 
• Include <dirent.h>: The d_name field contains the name of the file in the context of the directory it traverses. So it doesn't contain any path, just the name.

Introduction 

3. Process

Gesture recognition is a computing process that attempts to recognize and interpret human gestures using mathematical algorithms. Gesture recognition is not limited to just human hand gestures. Gesture recognition is a growing field of computer science, with an international conference devoted to gesture and facial recognition. As the field continues growing, there are many ways that it can be utilized. Gesture recognition computer processes are designed to enhance human-computer interaction, and can occur in multiple ways, such as use of touch screens and many more.

                             

3.1 Code:

It begins with the import of the libraries that would be used in the project. A vector was installed for the images, in addition to the variables of the trackbars and finally a counter for the gestures.

A vector string type function called loadFile was created which was used to load the images, which will be used to recognize a gesture.The second function called findMaxCountourn of type size t was created in order to find the maximum contour of our space, in which the gestures were performed. We create a type of void function to save new gestures and one for the track event, a double type function was created to calculate the Euclidean distance and finally a mat type function to calculate the histogram.

Finally, our last method, findImagePoints, was used to capture the points within the gesture that is being performed and then saved them as an image. Inside our main, we proceeded to activate the camera, created our trackbars, calculated the HU moments with their centroid, read the files and defined the screens to be displayed in the project, which in our case are 4: camera, ROI, Roi mask and M1.

4. Outcome

4.1 picture

4.2 picture

4.3 picture

4.4 Funcionamiento 

5. Conclusion

With the practice carried out in the detection of gestures through mimics, it is ideal for deaf and speaking people since they cannot perceive oral language in the same way as a person with full abilities. This type of practice can be implemented in schools for student learning. The great thing would be that they receive classes in a normal school and that by means of this technique; Through an electronic device, be it Tablet, cell phone or PC. This technique opens us to a field where starting from this base; We can achieve great things for this group of people in our society; An example would be to implement in a robot that recognizes mimic gestures and performs the order that is sent to it and is no longer just by voice.

    

 

sources:

1. https://dspace.ups.edu.ec/bitstream/123456789/19133/1/UPS-TTQ108.pdf

2. Kaehler, A., & Bradski, G. (2016). Learning OpenCV 3: computer vision in C++ with the OpenCV library. " O'Reilly Media, Inc.". 

3. Machio, V. D. (2014). Reconocimiento de Gestos Basado en Visión Estereoscópica y Redes Neuronales.http://tesis.blanque.com.ar/Home_files/Tesis_Victor_Machio.pdf