Abstract

     The aim of this paper is to develop an algorithm to enhance speech recognition of a stuttered speech. Stuttering is a disorder that affects the fluency of speech by involuntary repetition, prolongation of words/syllables, or involuntary silent intervals. Current speech recognition systems fail to recognize stuttered speech. Methods to detect stutter have been reported in literature but efficient techniques for stutter correction have not been reported. This paper addresses this issue and proposes methods to detect and correct stutter within acceptable time limits. To remove prolongation(s) from the sample, amplitude thresholding through neural networks is developed. Repetitions are removed through string repetition removal algorithm using an existing Text-to-Speech (TTS) system. Thus, the output signal, void of all stutters, produces better speech recognition.

Introduction

     Stuttering is a speech disorder characterized by repetition of sounds, syllables, or words; prolongation of sounds. An individual who faces this disorder knows what he or she intend to say but is unable to produce fluent speech. Millions of people, in today’s world, suffer from various speech disorders like stuttering, lisp, and articulation disorder. This often renders them unable to utilize certain things that a normal person takes for granted, like speech recognition systems.

     Stuttering disorder is characterized by disruptions in the production of speech sounds, called disfluencies. Disfluencies are not necessarily a problem; however, they can hinder communication when a person produces too many of them. Most people often produce brief disfluencies. For instance, some words are repeated or prolonged while others are preceded by an ‘um’ or ‘uh’. In most cases, stuttering has an impact at least on some daily activities. The specific everyday activities that a person finds challenging to perform, vary across individuals. For example, for some people, communication difficulties happen only during specific activities, like talking on the phone, talking before large groups, utilizing everyday tools that use speech as inputs. An author claims, “Stuttering cannot be permanently cured; it may go into remission for a time, or clients can learn to shape their speech into fluent speech with the appropriate speech pathology treatment” [1].

     The main objective of this paper is to present an algorithm that efficiently detects and corrects stutter in a speech segment of a person with stuttering speech disability. The proposed algorithm gives an accuracy level of 86% for 50 stutter speech samples. Two algorithms were used for more precise stutter removal system that can be built on any device.

     The developed system can be incorporated into any existing speech recognition system. It can also serve as a speech therapy system where a user suffering from stutter can sound like the correct output obtained from the system. Hence the device can be used by people suffering from stutter to use the existing virtual assistant services, or talk to others with confidence using the device. This would enhance the level of communication amongst people with this disorder.

1- INTRODUCTION

2- SINGLE-CHANNEL MULTI-TALKER SPEECH RECOGNITION

3- END-TO-END MULTI-TALKER SPEECH RECOGNITION

4- EXPERIMENT

5- CONCLUSION

REFERENCES

Abstract

Although significant progress has been made in singletalker automatic speech recognition (ASR), there is still a large performance gap between multi-talker and single-talker speech recognition systems. In this article, we propose an enhanced end-toend monaural multi-talker ASR architecture and training strategy to recognize the overlapped speech. The single-talker end-to-end model is extended to a multi-talker architecture with permutation invariant training (PIT). Several methods are designed to enhance the system performance, including speaker parallel attention, scheduled sampling, curriculum learning and knowledge distillation.More specifically, the speaker parallel attention extends the basic single shared attention module into multiple attention modules for each speaker, which can enhance the tracing and separation ability. Then the scheduled sampling and curriculum learning are proposed to make the model better optimized. Finally the knowledge distillation transfers the knowledge from an original single-speaker model to the current multi-speaker model in the proposed end-to-end multi-talker ASR structure. Our proposed architectures are evaluated and compared on the artificially mixed speech datasets generated from the WSJ0 reading corpus. The experiments demonstrate that our proposed architectures can significantly improve the multi-talker mixed speech recognition. The final system obtains more than 15% relative performance gains in both character error rate (CER) and word error rate (WER) compared to the basic end-to-end multi-talker ASR system.

INTRODUCTION

T HANKS to the advances in deep learning, automatic speech recognition (ASR) has achieved a huge progress. Deep neural networks (DNN) and hiddenMarkov model (HMM) based hybrid systems have achieved a very good performance, which are comparable with, or even surpassing, human performance [1]–[3]. Recently, there have been growing interests in developing end-to-end systems for speech recognition, in which multiple modules in the hybrid systems, such as the acoustic model (AM), lexicon model, and language model (LM), are folded into a single neural network model, so that they can be optimized simultaneously. Over the past few years, a variety of end-to-end (E2E) models have been proposed and they can be mainly categorized into connectionist temporal classification (CTC) based models [4], [5], and sequence to sequence (S2S) based models [6], [7]. The combined mode with both CTC and S2S [8] is also designed to further improve the end-to-end ASR system. The end-to-end systems have shown promising results according to existing works [8]–[10]. On the other hand, although a huge progress has been achieved on ASR, the current systems mainly focus on single-talker speech, and there is still a large performance gap between single-talker and multi-talker speech recognition. Processing the multi-talker mixed speech is a key problem when multi-talker mixed speech commonly exists in the complex real-world conditions, especially under the cocktail party scenarios [11]–[13].

Abstract

The spectral-based features, typically used in Automatic Speech Recognition (ASR) systems, reject the phase information of speech signals. Thus, employing extra features, in which the phase of the signal is not rejected, may fill this gap. Embedding the speech signal in the Reconstructed Phase Space (RPS) and then extracting some useful features from it, is a recently considered approach in this field. In this paper, we will follow this approach by evaluating some useful features from the Recurrence Plot (RP) of the embedded speech signals in the RPS; the proposed features are evaluated via applying a two-dimensional wavelet transform to the resulted RP diagrams. The proposed features are examined in an ASR task alone and in combination with the traditional Mel-Frequency Cepstral Coefficients (MFCC). For the second case, using English TIMIT corpus, 3.94% absolute classification accuracy improvement in the phoneme recognition accuracy rate, against using only the MFCC features is gained.

Introduction

In recent decades, a variety of linear models for speech coding, synthesis, and recognition with acceptable performances have been introduced. In this way, many types of research achieved improvement in the field of speech recognition by employing novel methods [1,2] or the detection of mispronunciation using Hidden Markov model [3]; however, there are nonlinear aerodynamic phenomena in the human speech production system which generally could not be included in linear models [4]. Therefore, nonlinear methods could potentially provide effective computational models to extract acoustic features which are useful for the nonlinear phenomena detection [4]. Furthermore, some recent studies have shown that utilizing nonlinear characteristics may improve the performances of the ASR systems [5].

Usual ASR systems exploit frequency domain features like Mel frequency cepstral coefficients [6]. The traditional frequency domain methods typically extract only the first and second order properties from the spectral patterns of speech signals [7]. However, there are many signals produced via nonlinear differential equations that have wide spectral characteristics [8]. In such cases, the frequency domain techniques are deficient, because it is impossible to dissociate the information of such a signal only in the frequency domain [5].

1- Introduction

2- Related work

3- Deep learning: An overview

4- Proposed speech emotion recognition system

5- Experimental setup

6- Experiments and results

7- Discussion

8- Conclusion

References

Abstract

Speech Emotion Recognition (SER) can be regarded as a static or dynamic classification problem, which makes SER an excellent test bed for investigating and comparing various deep learning architectures. We describe a frame-based formulation to SER that relies on minimal speech processing and end-to-end deep learning to model intra-utterance dynamics. We use the proposed SER system to empirically explore feed-forward and recurrent neural network architectures and their variants. Experiments conducted illuminate the advantages and limitations of these architectures in paralinguistic speech recognition and emotion recognition in particular. As a result of our exploration, we report state-of-the-art results on the IEMOCAP database for speaker-independent SER and present quantitative and qualitative assessments of the models’ performances.

Introduction

In recent years, deep learning in neural networks has achieved tremendous success in various domains that led to multiple deep learning architectures emerging as effective models across numerous tasks. Feed-forward architectures such as Deep Neural Networks (DNNs) and Convolutional Neural Networks (ConvNets) have been particularly successful in image and video processing as well as speech recognition, while recurrent architectures such as Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) RNNs have been effective in speech recognition and natural language processing (LeCun, Bengio, & Hinton, 2015; Schmidhuber, 2015). These architectures process and model information in different ways and have their own advantages and limitations. For instance, ConvNets are able to deal with high-dimensional inputs and learn features that are invariant to small variations and distortions (Krizhevsky, Sutskever, & Hinton, 2012), whereas LSTM-RNNs are able to deal with variable length inputs and model sequential data with long range context (Graves, 2008). In this paper, we investigate the application of end-to-end deep learning to Speech Emotion Recognition (SER) and critically explore how each of these architectures can be employed in this task. ∗ Corresponding author. E-mail addresses: haytham.fayek@ieee.org (H.M. Fayek), margaret.lech@rmit.edu.au (M. Lech), lawrence.cavedon@rmit.edu.au (L. Cavedon). SER can be regarded as a static or dynamic classification problem, which has motivated two popular formulations in the literature to the task (Ververidis & Kotropoulos, 2006): turn-based processing (also known as static modeling), which aims to recognize emotions from a complete utterance; or frame-based processing (also known as dynamic modeling), which aims to recognize emotions at the frame level. In either formulation, SER can be employed in stand-alone applications; e.g. emotion monitoring, or integrated into other systems for emotional awareness; e.g. integrating SER into Automatic Speech Recognition (ASR) to improve its capability in dealing with emotional speech (Cowie et al., 2001; Fayek, Lech, & Cavedon, 2016b; Fernandez, 2004). Frame-based processing is more robust since it does not rely on segmenting the input speech into utterances and can model intra-utterance emotion dynamics (Arias, Busso, & Yoma, 2013; Fayek, Lech, & Cavedon, 2015). However, empirical comparisons between frame-based processing and turn-based processing in prior work have demonstrated the superiority of the latter (Schuller, Vlasenko, Eyben, Rigoll, & Wendemuth, 2009; Vlasenko, Schuller, Wendemuth, & Rigoll, 2007).

Sequential data often emerge in the measurement of time series, e.g. the acoustic features at the time frames of speech signals [1] and kinematic and biological signals in human movements [2]. Hence, sequential pattern recognition is increasingly used in many data analysis fields such as handwriting recognition [3,4], speech recognition [5,6], activity recognition [7,8], and gesture recognition [2]. In this approach, sequences of information are utilized in decision-making [9]. Sequential pattern recognition in time-series information matches the input data with a pattern, both of which being sequences of information. However, inequality in the length of sequences between the input data and the pattern is a typical problem. Studies in 1960s showed that an appropriate pattern recognition tool should be developed to overcome this problem in speech recognition [10]. One of the main solutions to this problem is the “time-wrapping” idea, presented by Vintsyk [11]. He showed how dynamic programming can be used to find the best assignment between two sequences. In 1970s and 1980s, many researchers proposed various models that mostly included modeling of acoustic information in a network with some finite states, deriving the stochastic information of acoustic features and comparing them with the input signal using the dynamic time-wrapping method [12]. Some examples of these works can be seen in [13–16]. The most well-known method in this field is Hidden Markov Model (HMM) [17]. The capabilities of the HMM such as the ability to match with stochastic sequential time series and suitable training algorithms have made it a pervasive tool in speech recognition. Since mid-1990s, the HMM has been a dominant method in handwriting and speech recognition [18]. Besides, many researchers have demonstrated its superiority to short time classifiers such as MLP [19] and SVM [8]. Despite the powerful mathematical modeling of the HMMs, the main criticisms about them fall into two categories: 1) limitations due to the selected mathematical modeling assumptions.