Abstract

     Depression is a serious mental health condition that may lead to poor mental and emotional functioning at work, at school and in the family causing the mental imbalance. In worst scenarios, depression may lead to severe anxiety or suicide. Hence, it is necessary to diagnose depression at early stages. This paper elaborates the development of a novel approach for a convolutional neural network model that can examine facial images from the recorded interview sessions to discover facial patterns that could indicate depression level. The user-generated data helps to distinguish between different depressive groups with depression symptoms that can manifest people with various mental illnesses in different ways. In particular, we want to automatically predict the depression scale and differentiate depression from other mental disorders using the patient’s psychiatric illness history and dynamic textual descriptions extracted from the user inputs. We apply the k-nearest neighbour algorithm on the dynamic textual descriptors to make a linguistic analysis for classifying mental illness into different classes. We apply dimensionality reduction and regression using the Random Forest algorithm to predict the depression scale. The proposed framework is an extension to pre-existing frameworks, replacing the handcrafted feature extraction technique with the deep feature extraction. The model performs 2.7% better than existing frameworks in facial detection and feature extraction.

Introduction

     In India, depression is common in adults between the ages of 16–25, considered to be a leading cause of disability. For years, researchers have to identify and map the relationship between brain function and structure using neuroimaging data. The researchers at the University of Texas have identified a unique technique to categorize people susceptible to developing depression and anxiety using deep learning with a supercomputer. Now they are using the Stampede supercomputer at the Texas advanced computing centre to train deep learning algorithms that can identify similarities among hundreds of patients using magnetic resonance imaging genomics data and other factors to predict patients at risk of depression and psychological disorders (Valstar et al., 2013). In the past, the researchers have worked on the development of a model that takes raw text and audio segments as input and analyzes the wave-forms that predict depression. The work provides a method for deep learning-based segmentation to detect depression, as well as irregular segmentation and masks used for Gabor wavelength detection (Long et al, 2014). A Gabor filter, named after Dennis Gabor, is a linear local texture filter in image processing. It analyses if the model includes any specific frequency content in specific directions in a localized region around the point or region of analysis. The researchers trained the deep learning algorithm using extracted audio and textual features from clinical patients with suicide attempts and higher rate of mood swings (Ma et al, 2016). Extreme Gradient boosting technique is used for identifying and categorizing the important parameters of depression and predicting depression cases by re-sampling methods using different balanced samples but the researchers were not able to precisely predict the depression scale. Also differentiating the mental disorders is a tedious task due to similarities in symptoms (Marcus et al., 2012).

Conclusion and future work

     Here, we have developed an AI-based automated system for predicting the depression scale. The model is purely based on recognizing facial expressions from recorded video sessions of depressed patients. We have used deep learning algorithms for extracting dynamic facial features and recognizing expressions producing the feature dynamic history histogram applied to feature dynamic vector sequences applied to the video sample. After calculating the feature dynamic history histogram, we apply regression techniques to correlate between feature and depression scales. The experimental results showed that the model had performed well on the test video dataset. The deep feature extraction technique significantly performs much better than the handcrafted feature extraction technique. The feature attributes are directly extracted from the responses provided by the convolutional layers perform best rather the performance of the neural networks are closely connected giving benchmarking results.

Recently, the study of emotion recognition has received increasing attentions by the rapid development of noninvasive sensor technologies, machine learning algorithms and compute capability of computers. Compared with single modal emotion recognition, the multimodal paradigm introduces complementary information for emotion recognition. Hence, in this work, we presented a decision level fusion framework for detecting emotions continuously by fusing the Electroencephalography (EEG) and facial expressions. Three types of movie clips (positive, negative, and neutral) were utilized to elicit specific emotions of subjects, the EEG and facial expression signals were recorded simultaneously. The power spectrum density (PSD) features of EEG were extracted by time-frequency analysis, and then EEG features were selected for regression. For the facial expression, the facial geometric features were calculated by facial landmark localization. Long short-term memory networks (LSTM) were utilized to accomplish the decision level fusion and captured temporal dynamics of emotions. The results have shown that the proposed method achieved outstanding performance for continuous emotion recognition, and it yields 0.625±0.029 of concordance correlation coefficient (CCC). From the results, the fusion of two modalities outperformed EEG and facial expression separately. Furthermore, different numbers of time-steps of LSTM was applied to analyze the temporal dynamic capturing.

Introduction

Emotion is a psychophysiological process of perception and cognition to object or situation, and it plays an important role in human-human natural communication. However, the emotion recognition was neglected in the field of human-computer interaction (HCI). Due to the explosion of machine learning in the cognitive science, affective computing has emerged to integrate emotion recognition into HCI. Nowadays, emotion recognition system aims to establish a harmonious HCI by endowing computers with the ability to recognize, understand, express and adapt to human emotions [1]. Hence, it provides potentially applications for emotion recognition in many fields, such as human robot interaction (HRI) [2], safe driving [3], social networking [4] and distance education [5]. These applications manifested the different modalities interaction which provides complementary information to improve the precision and robust of the emotion recognition system. In order to represent emotions, the discrete models and dimensional models were proposed by psychologists [6]. The discrete emotion models turned emotion recognition into classification problem. Six basic emotions (happiness, anger, sadness, surprise, fear and disgust) can be recognized as prototypes from which other emotions are derived [7]. However, the emotions expressed in communication are complex, and one basic emotion can hardly describe the human feeling under certain situation. In addition, emotions could be mapped in multi-dimensional spaces that could maximize the largest variance of all the possible emotions [8]. The valence-arousal plane is one of the well-known dimensional models of emotion that maps the emotions into a two-dimensional circular space.

Emotion Detection today is one of the most important directions for research papers in any specific field. It performs a very important role in organizing human-computer interaction. We as humans are able to understand the emotions of other person but it is impossible for the computer to do so. The current work is to meet the same as accurately as possible. Emotion detection can do either through textual content, speech, facial expression or features [1]. EEG is a test that measures and records the electrical activity of our brain using special sensors [2]. The computer records the brain electrical activity as wavy lines. Positive situations, such as seizures, possible to seen by the changes in the everyday pattern of the brain’s electrical activity. EEG helmet uses a set of sensors to measure the signal changes generated by the brain to discover people’s thoughts, feelings and expressions and connects wirelessly to the personal computer, It has 14 sensors which a put on particular places on the scalp to give acceptable detection accuracy [3]. Human emotion is very hard to find simply through looking on the face and additionally the conduct of someone [4]. This research paper conducted to discover or perceive human emotion through the take a look at the of brain waves. Additionally, the research paper aims to improve software program that can detect human emotions fast. This study aims at EEG signals of relationship and human feelings. Brain waves produced by a human to change from one person to another depending on their current activities. For instance, the brain waves of the relaxed person are different from the waves of the person doing the hard work and bad mood. These brain waves classified into five classification indicating various conditions which Delta, theta, alpha, beta, and gamma [1]. In our research paper, we mainly try to discover the relationship between EEG and human emotions. Instead of using the face directly, we apply a dynamic programming to extract the maximum number of quality service that provides to the user when the device captures specific signals for each emotion. The aim of this operation is to decrease the impact of factors irrelevant to emotion. In order to discover the features which are most related to emotion, we reduce feature measure through correlation degree whilst keeping a stable classification execution, and figure out the brain areas and frequency constraint according to with the positions to which they exposed. In addition to, we try to find the track of emotion changes with multiple models. The structure of this paper as follows. In Section II we present some related works. In Section III show the methodology and experimental design. In Section IV illustrates the overview of the results and discussion of this present work. Finally, Section V summarizes the conclusions and briefly highlights future work.

According to the 2016 third quarter earnings report [1] released by Sina Weibo, as of September 30, 2016, the monthly number of active users on Weibo has reached 297 million. In September, 2016, the number of active users has reached 132 million, representing an increase of 32% over the same period last year. Micro-blog in the video, travel, sports and other fields have been further developed. In 2016, most ofthe active users on micro-blog are highly educated, they are the main force of micro-blog, accounting for up to 77.8%, and their emotional states are often characterized by the microblogs that they released. Sina Weibo has a large number of young users, and they are an important part of the main consumer and society. User emotion modeling and anomaly detection on micro-blog is an important field of emotional analysis, which can help the enterprises to make business decisions, help the government to monitor public opinion and public safety through social network, prevent the spread of irrational emotions in social network or even in real world, respond timely to the possible negative incidents to prevent some criminals who attempts to spread rumors [2] through micro-blog of high-precision emotional analysis system. Zhang [9] built an emotional dictionary based on the emotional words and phrases commonly used of emotional factors to recognize and classify the emotion on micro-blog, which achieved good results. Zhao [10] considered the object of a text to improve the emotional classifi- cation accuracy to detect the social anomaly, the Twitter text were chose as the sample for testing, by comparing the proportion of negative emotions to observe anomaly in a day, the conclusions are general and could not accurately analyze the specific abnormal event or user. Li [11], who was based on real-time event monitoring framework and system of micro-blog, proposed a rule-based and statistical method, used time series modelto monitor anomaly, which proved more effective than the ordinary model. Yin [12] proposed a micro-blog anomaly ranking detection method based on the lifting coefficient, which effectively prevented the artificial manipulation to improve the ranking of micro-blog. Experiments onthe simulationdata set showedthatthemethodcouldeffectively identify micro-blog anomaly ranking by micro-blog topology. Anomaly detection methods mentioned above are mainly based on dictionary, text, neural network, time series, statistic, rule and rank, which require a large number of annotated corpus, but the annotations workload are really heavy. In addition, the current methods tend to classify and analyze all the data on a social platform to detect outbreaks or abnormal events from the time aspect, but there is little research on the detection of abnormal emotion for the individual user.

To understand and convey each other’s intentions in a natural way, humancomputer interaction (HCI) has been paid more attentions in recent years [1]. The primary problem that the HCI faces is how to master the ability of identifying emotional information accurately, which is similar to the emotional intelligence capability in human-robot interaction [2]. As a fast and easyunderstand way for communication, speech signal was considered to identify emotions [3]. It is believed that speech conveys not only syntactic and semantic contents of the linguistic sentences but also the emotional states of humans [4]. Thus, human emotion recognition using speech signal is feasible, which mainly studies on how to identify the emotional or physical states of humans from his/her voice automatically [5]. In speech emotion recognition, one of the central research issues is how to select an optimal feature set from speech signals [6]. Most of the previous works on speech emotion recognition have been devoted on the analysis of speech prosodic features and spectral information [7]. And some novel feature parameters are used for speech emotion recognition, such as the Fourier parameters [8]. Although there are many acoustic parameters have been proven to contain emotional information, little success has been achieved in realizing such a set of features that performs consistently over different conditions [9]. Thus, most researchers prefer to use mixing feature set that is composed of many kinds of features containing more emotional information [10]. However, using mixing feature set may give rise to high dimension and redundancy of speech features, thereby it makes the learning process complicated for most machine learning algorithms and increases the likelihood of overfitting. Therefore, feature selection is indispensible to reduce the dimensions redundancy of features.