Sign language recognition is often carried out using hierarchical classification approach to reduce complexity and enhance accuracy. In this paper, mutli-label classification is proposed for categorization of a sign based on its lexical attributes followed by final classification of the sign. Results are presented for classification of 100 isolated signs from the Indian sign language recorded using multiple surface electromyogram and inertial measurement units on both the forearms of 10 different signers. Signals from both the hands are processed in an integrated manner to identify static or dynamic state of the two hands. Moreover, symmetry in the motion of two hands is also utilized for sign categorization using novel features. In the classic tree-based categorization of signs, there is error propagation, which results in a classification error of 6.22%. Whereas in the proposed mutli-label classification approach, error propagation is avoided and the average classification error of 2.73% is observed.

1. Introduction

A language provides humans with a structured means to exchange information with each other. While languages like Hindi and English use verbal or written mode of communication, sign languages, on the other hand, involve the use of visual gestures and signs. People with hearing and speech disabilities can communicate more naturally in sign language as compared to verbal languages. However, since most people do not understand sign languages, there is often a communication barrier experienced by a person wishing to converse in sign language. The use of a human interpreter or written form of communication is not always convenient. According to the Census 2001 of the Ministry of Home Affairs [1], there are around 1.26 million deaf people and around 1.64 million people with speech disability in India, while there are only 250 certified sign language interpreters in India. An electronic, wearable sign language recognition system shall be very useful in reducing the communication barrier that exists between a signer and a non-signer. There are several challenges in designing such a system.

SIGN language plays a vital role for deaf and mute people to communicate among themselves or with normal people in a non-verbal manner. Gestures are the primary method to convey messages, which are usually conducted in a three-dimensional space, known as a signing space [1], through an integration of manual and non-manual signals. Manual signals commonly correspond to hand motions and hand posturing, whereas non-manual signals correspond to an external appearance such as mouth movements, facial expressions, and body orientation [2]. Nevertheless, sign language has not been standardized globally. Each nation has developed its own sign language, such as the American Sign Language (ASL) and Germany Sign Language (GSL). However, each sign language varies slightly within different regions of the same country. Hence, it can be a challenge to develop a standardized sign language interpretation system for use worldwide. In a previous study, sign languages have been recognized using two major techniques, i.e., vision and non-vision approaches [3]. In fact, vision method is the major technique applied for sign recognition in the past decades. A system that uses a camera to observe the information obtained through hand and finger motions is the most widely adopted visual-based approach [4]. Tremendous effort and study has gone into the development of vision-based sign recognition systems worldwide. Indeed, vision-based gesture recognition systems can be subdivided into direct and indirect approaches. A direct approach detects the hand gestures based on the RGB color spaces of the skin color. For instance, Goyal et al. [5] identified Indian Sign Language (ISL) using the scale invariance Fourier transform (SIFT) algorithm by searching the matched key points between the input image and images stored in a database. A similar method was also applied by More et al. [6] using the SIFT algorithm, which further reduces the dimensions of the feature vector using a principal component analysis (PCA) algorithm for speeding up the processing time. To detect the dynamic hand gestures used in Japanese Sign Language (JSL), Murakami et al. [7] proposed the use of recurrent neural networks capable of recognizing the JSL finger alphabet, which has 42 symbols. In contrast, Chowdary et al. [8] used a simple scanning method to compute the orientation and movement of fingers in binary converted images captured from a web camera. Khan et al. [9] proposed a more sophisticated gesture recognition system using digital images, including image filtering (pre-processing), image segmentation, color segmentation, skin detection (finger and hand detection using binary images), and template matching. Meanwhile, an indirect approach identifies the fingers and hand gestures based on the RGB color spaces segmented based on different colors for each finger using a data glove.