Word speech recognition methods

Speech recognition is an interdisciplinary subfield of computer science and computational linguistics that develops methodologies and technologies that enable the recognition and translation of spoken language into text by computers with the main benefit of searchability. It is also known as automatic speech recognition (ASR), computer speech recognition or speech to text (STT). It incorporates knowledge and research in the computer science, linguistics and computer engineering fields. The reverse process is speech synthesis.

Some speech recognition systems require «training» (also called «enrollment») where an individual speaker reads text or isolated vocabulary into the system. The system analyzes the person’s specific voice and uses it to fine-tune the recognition of that person’s speech, resulting in increased accuracy. Systems that do not use training are called «speaker-independent»^[1] systems. Systems that use training are called «speaker dependent».

Speech recognition applications include voice user interfaces such as voice dialing (e.g. «call home»), call routing (e.g. «I would like to make a collect call»), domotic appliance control, search key words (e.g. find a podcast where particular words were spoken), simple data entry (e.g., entering a credit card number), preparation of structured documents (e.g. a radiology report), determining speaker characteristics,^[2] speech-to-text processing (e.g., word processors or emails), and aircraft (usually termed direct voice input).

The term voice recognition^[3][4][5] or speaker identification^[6][7][8] refers to identifying the speaker, rather than what they are saying. Recognizing the speaker can simplify the task of translating speech in systems that have been trained on a specific person’s voice or it can be used to authenticate or verify the identity of a speaker as part of a security process.

From the technology perspective, speech recognition has a long history with several waves of major innovations. Most recently, the field has benefited from advances in deep learning and big data. The advances are evidenced not only by the surge of academic papers published in the field, but more importantly by the worldwide industry adoption of a variety of deep learning methods in designing and deploying speech recognition systems.

History[edit]

The key areas of growth were: vocabulary size, speaker independence, and processing speed.

Pre-1970[edit]

• 1952 – Three Bell Labs researchers, Stephen Balashek,^[9] R. Biddulph, and K. H. Davis built a system called «Audrey»^[10] for single-speaker digit recognition. Their system located the formants in the power spectrum of each utterance.^[11]
• 1960 – Gunnar Fant developed and published the source-filter model of speech production.
• 1962 – IBM demonstrated its 16-word «Shoebox» machine’s speech recognition capability at the 1962 World’s Fair.^[12]
• 1966 – Linear predictive coding (LPC), a speech coding method, was first proposed by Fumitada Itakura of Nagoya University and Shuzo Saito of Nippon Telegraph and Telephone (NTT), while working on speech recognition.^[13]
• 1969 – Funding at Bell Labs dried up for several years when, in 1969, the influential John Pierce wrote an open letter that was critical of and defunded speech recognition research.^[14] This defunding lasted until Pierce retired and James L. Flanagan took over.

Raj Reddy was the first person to take on continuous speech recognition as a graduate student at Stanford University in the late 1960s. Previous systems required users to pause after each word. Reddy’s system issued spoken commands for playing chess.

Around this time Soviet researchers invented the dynamic time warping (DTW) algorithm and used it to create a recognizer capable of operating on a 200-word vocabulary.^[15] DTW processed speech by dividing it into short frames, e.g. 10ms segments, and processing each frame as a single unit. Although DTW would be superseded by later algorithms, the technique carried on. Achieving speaker independence remained unsolved at this time period.

1970–1990[edit]

• 1971 – DARPA funded five years for Speech Understanding Research, speech recognition research seeking a minimum vocabulary size of 1,000 words. They thought speech understanding would be key to making progress in speech recognition, but this later proved untrue.^[16] BBN, IBM, Carnegie Mellon and Stanford Research Institute all participated in the program.^[17][18] This revived speech recognition research post John Pierce’s letter.

• 1972 – The IEEE Acoustics, Speech, and Signal Processing group held a conference in Newton, Massachusetts.
• 1976 – The first ICASSP was held in Philadelphia, which since then has been a major venue for the publication of research on speech recognition.^[19]

During the late 1960s Leonard Baum developed the mathematics of Markov chains at the Institute for Defense Analysis. A decade later, at CMU, Raj Reddy’s students James Baker and Janet M. Baker began using the Hidden Markov Model (HMM) for speech recognition.^[20] James Baker had learned about HMMs from a summer job at the Institute of Defense Analysis during his undergraduate education.^[21] The use of HMMs allowed researchers to combine different sources of knowledge, such as acoustics, language, and syntax, in a unified probabilistic model.

• By the mid-1980s IBM’s Fred Jelinek’s team created a voice activated typewriter called Tangora, which could handle a 20,000-word vocabulary^[22] Jelinek’s statistical approach put less emphasis on emulating the way the human brain processes and understands speech in favor of using statistical modeling techniques like HMMs. (Jelinek’s group independently discovered the application of HMMs to speech.^[21]) This was controversial with linguists since HMMs are too simplistic to account for many common features of human languages.^[23] However, the HMM proved to be a highly useful way for modeling speech and replaced dynamic time warping to become the dominant speech recognition algorithm in the 1980s.^[24]

• 1982 – Dragon Systems, founded by James and Janet M. Baker,^[25] was one of IBM’s few competitors.

Practical speech recognition[edit]

The 1980s also saw the introduction of the n-gram language model.

• 1987 – The back-off model allowed language models to use multiple length n-grams, and CSELT^[26] used HMM to recognize languages (both in software and in hardware specialized processors, e.g. RIPAC).

Much of the progress in the field is owed to the rapidly increasing capabilities of computers. At the end of the DARPA program in 1976, the best computer available to researchers was the PDP-10 with 4 MB ram.^[23] It could take up to 100 minutes to decode just 30 seconds of speech.^[27]

Two practical products were:

• 1984 – was released the Apricot Portable with up to 4096 words support, of which only 64 could be held in RAM at a time.^[28]
• 1987 – a recognizer from Kurzweil Applied Intelligence
• 1990 – Dragon Dictate, a consumer product released in 1990^[29][30] AT&T deployed the Voice Recognition Call Processing service in 1992 to route telephone calls without the use of a human operator.^[31] The technology was developed by Lawrence Rabiner and others at Bell Labs.

By this point, the vocabulary of the typical commercial speech recognition system was larger than the average human vocabulary.^[23] Raj Reddy’s former student, Xuedong Huang, developed the Sphinx-II system at CMU. The Sphinx-II system was the first to do speaker-independent, large vocabulary, continuous speech recognition and it had the best performance in DARPA’s 1992 evaluation. Handling continuous speech with a large vocabulary was a major milestone in the history of speech recognition. Huang went on to found the speech recognition group at Microsoft in 1993. Raj Reddy’s student Kai-Fu Lee joined Apple where, in 1992, he helped develop a speech interface prototype for the Apple computer known as Casper.

Lernout & Hauspie, a Belgium-based speech recognition company, acquired several other companies, including Kurzweil Applied Intelligence in 1997 and Dragon Systems in 2000. The L&H speech technology was used in the Windows XP operating system. L&H was an industry leader until an accounting scandal brought an end to the company in 2001. The speech technology from L&H was bought by ScanSoft which became Nuance in 2005. Apple originally licensed software from Nuance to provide speech recognition capability to its digital assistant Siri.^[32]

2000s[edit]

In the 2000s DARPA sponsored two speech recognition programs: Effective Affordable Reusable Speech-to-Text (EARS) in 2002 and Global Autonomous Language Exploitation (GALE). Four teams participated in the EARS program: IBM, a team led by BBN with LIMSI and Univ. of Pittsburgh, Cambridge University, and a team composed of ICSI, SRI and University of Washington. EARS funded the collection of the Switchboard telephone speech corpus containing 260 hours of recorded conversations from over 500 speakers.^[33] The GALE program focused on Arabic and Mandarin broadcast news speech. Google’s first effort at speech recognition came in 2007 after hiring some researchers from Nuance.^[34] The first product was GOOG-411, a telephone based directory service. The recordings from GOOG-411 produced valuable data that helped Google improve their recognition systems. Google Voice Search is now supported in over 30 languages.

In the United States, the National Security Agency has made use of a type of speech recognition for keyword spotting since at least 2006.^[35] This technology allows analysts to search through large volumes of recorded conversations and isolate mentions of keywords. Recordings can be indexed and analysts can run queries over the database to find conversations of interest. Some government research programs focused on intelligence applications of speech recognition, e.g. DARPA’s EARS’s program and IARPA’s Babel program.

In the early 2000s, speech recognition was still dominated by traditional approaches such as Hidden Markov Models combined with feedforward artificial neural networks.^[36]
Today, however, many aspects of speech recognition have been taken over by a deep learning method called Long short-term memory (LSTM), a recurrent neural network published by Sepp Hochreiter & Jürgen Schmidhuber in 1997.^[37] LSTM RNNs avoid the vanishing gradient problem and can learn «Very Deep Learning» tasks^[38] that require memories of events that happened thousands of discrete time steps ago, which is important for speech.
Around 2007, LSTM trained by Connectionist Temporal Classification (CTC)^[39] started to outperform traditional speech recognition in certain applications.^[40] In 2015, Google’s speech recognition reportedly experienced a dramatic performance jump of 49% through CTC-trained LSTM, which is now available through Google Voice to all smartphone users.^[41] Transformers, a type of neural network based on solely on attention, have been widely adopted in computer vision^[42][43] and language modeling,^[44][45] sparking the interest of adapting such models to new domains, including speech recognition.^[46][47][48] Some recent papers reported superior performance levels using transformer models for speech recognition, but these models usually require large scale training datasets to reach high performance levels.

The use of deep feedforward (non-recurrent) networks for acoustic modeling was introduced during the later part of 2009 by Geoffrey Hinton and his students at the University of Toronto and by Li Deng^[49] and colleagues at Microsoft Research, initially in the collaborative work between Microsoft and the University of Toronto which was subsequently expanded to include IBM and Google (hence «The shared views of four research groups» subtitle in their 2012 review paper).^[50][51][52] A Microsoft research executive called this innovation «the most dramatic change in accuracy since 1979».^[53] In contrast to the steady incremental improvements of the past few decades, the application of deep learning decreased word error rate by 30%.^[53] This innovation was quickly adopted across the field. Researchers have begun to use deep learning techniques for language modeling as well.

In the long history of speech recognition, both shallow form and deep form (e.g. recurrent nets) of artificial neural networks had been explored for many years during 1980s, 1990s and a few years into the 2000s.^[54][55][56]
But these methods never won over the non-uniform internal-handcrafting Gaussian mixture model/Hidden Markov model (GMM-HMM) technology based on generative models of speech trained discriminatively.^[57] A number of key difficulties had been methodologically analyzed in the 1990s, including gradient diminishing^[58] and weak temporal correlation structure in the neural predictive models.^[59][60] All these difficulties were in addition to the lack of big training data and big computing power in these early days. Most speech recognition researchers who understood such barriers hence subsequently moved away from neural nets to pursue generative modeling approaches until the recent resurgence of deep learning starting around 2009–2010 that had overcome all these difficulties. Hinton et al. and Deng et al. reviewed part of this recent history about how their collaboration with each other and then with colleagues across four groups (University of Toronto, Microsoft, Google, and IBM) ignited a renaissance of applications of deep feedforward neural networks to speech recognition.^{[51][52][61][62]}

2010s[edit]

By early 2010s speech recognition, also called voice recognition^[63][64][65] was clearly differentiated from speaker recognition, and speaker independence was considered a major breakthrough. Until then, systems required a «training» period. A 1987 ad for a doll had carried the tagline «Finally, the doll that understands you.» – despite the fact that it was described as «which children could train to respond to their voice».^[12]

In 2017, Microsoft researchers reached a historical human parity milestone of transcribing conversational telephony speech on the widely benchmarked Switchboard task. Multiple deep learning models were used to optimize speech recognition accuracy. The speech recognition word error rate was reported to be as low as 4 professional human transcribers working together on the same benchmark, which was funded by IBM Watson speech team on the same task.^[66]

Models, methods, and algorithms[edit]

Both acoustic modeling and language modeling are important parts of modern statistically based speech recognition algorithms. Hidden Markov models (HMMs) are widely used in many systems. Language modeling is also used in many other natural language processing applications such as document classification or statistical machine translation.

Hidden Markov models[edit]

Modern general-purpose speech recognition systems are based on hidden Markov models. These are statistical models that output a sequence of symbols or quantities. HMMs are used in speech recognition because a speech signal can be viewed as a piecewise stationary signal or a short-time stationary signal. In a short time scale (e.g., 10 milliseconds), speech can be approximated as a stationary process. Speech can be thought of as a Markov model for many stochastic purposes.

Another reason why HMMs are popular is that they can be trained automatically and are simple and computationally feasible to use. In speech recognition, the hidden Markov model would output a sequence of n-dimensional real-valued vectors (with n being a small integer, such as 10), outputting one of these every 10 milliseconds. The vectors would consist of cepstral coefficients, which are obtained by taking a Fourier transform of a short time window of speech and decorrelating the spectrum using a cosine transform, then taking the first (most significant) coefficients. The hidden Markov model will tend to have in each state a statistical distribution that is a mixture of diagonal covariance Gaussians, which will give a likelihood for each observed vector. Each word, or (for more general speech recognition systems), each phoneme, will have a different output distribution; a hidden Markov model for a sequence of words or phonemes is made by concatenating the individual trained hidden Markov models for the separate words and phonemes.

Described above are the core elements of the most common, HMM-based approach to speech recognition. Modern speech recognition systems use various combinations of a number of standard techniques in order to improve results over the basic approach described above. A typical large-vocabulary system would need context dependency for the phonemes (so phonemes with different left and right context have different realizations as HMM states); it would use cepstral normalization to normalize for a different speaker and recording conditions; for further speaker normalization, it might use vocal tract length normalization (VTLN) for male-female normalization and maximum likelihood linear regression (MLLR) for more general speaker adaptation. The features would have so-called delta and delta-delta coefficients to capture speech dynamics and in addition, might use heteroscedastic linear discriminant analysis (HLDA); or might skip the delta and delta-delta coefficients and use splicing and an LDA-based projection followed perhaps by heteroscedastic linear discriminant analysis or a global semi-tied co variance transform (also known as maximum likelihood linear transform, or MLLT). Many systems use so-called discriminative training techniques that dispense with a purely statistical approach to HMM parameter estimation and instead optimize some classification-related measure of the training data. Examples are maximum mutual information (MMI), minimum classification error (MCE), and minimum phone error (MPE).

Decoding of the speech (the term for what happens when the system is presented with a new utterance and must compute the most likely source sentence) would probably use the Viterbi algorithm to find the best path, and here there is a choice between dynamically creating a combination hidden Markov model, which includes both the acoustic and language model information and combining it statically beforehand (the finite state transducer, or FST, approach).

A possible improvement to decoding is to keep a set of good candidates instead of just keeping the best candidate, and to use a better scoring function (re scoring) to rate these good candidates so that we may pick the best one according to this refined score. The set of candidates can be kept either as a list (the N-best list approach) or as a subset of the models (a lattice). Re scoring is usually done by trying to minimize the Bayes risk^[67] (or an approximation thereof): Instead of taking the source sentence with maximal probability, we try to take the sentence that minimizes the expectancy of a given loss function with regards to all possible transcriptions (i.e., we take the sentence that minimizes the average distance to other possible sentences weighted by their estimated probability). The loss function is usually the Levenshtein distance, though it can be different distances for specific tasks; the set of possible transcriptions is, of course, pruned to maintain tractability. Efficient algorithms have been devised to re score lattices represented as weighted finite state transducers with edit distances represented themselves as a finite state transducer verifying certain assumptions.^[68]

Dynamic time warping (DTW)-based speech recognition[edit]

Dynamic time warping is an approach that was historically used for speech recognition but has now largely been displaced by the more successful HMM-based approach.

Dynamic time warping is an algorithm for measuring similarity between two sequences that may vary in time or speed. For instance, similarities in walking patterns would be detected, even if in one video the person was walking slowly and if in another he or she were walking more quickly, or even if there were accelerations and deceleration during the course of one observation. DTW has been applied to video, audio, and graphics – indeed, any data that can be turned into a linear representation can be analyzed with DTW.

A well-known application has been automatic speech recognition, to cope with different speaking speeds. In general, it is a method that allows a computer to find an optimal match between two given sequences (e.g., time series) with certain restrictions. That is, the sequences are «warped» non-linearly to match each other. This sequence alignment method is often used in the context of hidden Markov models.

Neural networks[edit]

Neural networks emerged as an attractive acoustic modeling approach in ASR in the late 1980s. Since then, neural networks have been used in many aspects of speech recognition such as phoneme classification,^[69] phoneme classification through multi-objective evolutionary algorithms,^[70] isolated word recognition,^[71] audiovisual speech recognition, audiovisual speaker recognition and speaker adaptation.

Neural networks make fewer explicit assumptions about feature statistical properties than HMMs and have several qualities making them attractive recognition models for speech recognition. When used to estimate the probabilities of a speech feature segment, neural networks allow discriminative training in a natural and efficient manner. However, in spite of their effectiveness in classifying short-time units such as individual phonemes and isolated words,^[72] early neural networks were rarely successful for continuous recognition tasks because of their limited ability to model temporal dependencies.

One approach to this limitation was to use neural networks as a pre-processing, feature transformation or dimensionality reduction,^[73] step prior to HMM based recognition. However, more recently, LSTM and related recurrent neural networks (RNNs),^{[37][41][74][75]} Time Delay Neural Networks(TDNN’s),^[76] and transformers.^[46][47][48] have demonstrated improved performance in this area.

Deep feedforward and recurrent neural networks[edit]

Deep Neural Networks and Denoising Autoencoders^[77] are also under investigation. A deep feedforward neural network (DNN) is an artificial neural network with multiple hidden layers of units between the input and output layers.^[51] Similar to shallow neural networks, DNNs can model complex non-linear relationships. DNN architectures generate compositional models, where extra layers enable composition of features from lower layers, giving a huge learning capacity and thus the potential of modeling complex patterns of speech data.^[78]

A success of DNNs in large vocabulary speech recognition occurred in 2010 by industrial researchers, in collaboration with academic researchers, where large output layers of the DNN based on context dependent HMM states constructed by decision trees were adopted.^{[79][80]
[81]} See comprehensive reviews of this development and of the state of the art as of October 2014 in the recent Springer book from Microsoft Research.^[82] See also the related background of automatic speech recognition and the impact of various machine learning paradigms, notably including deep learning, in
recent overview articles.^[83][84]

One fundamental principle of deep learning is to do away with hand-crafted feature engineering and to use raw features. This principle was first explored successfully in the architecture of deep autoencoder on the «raw» spectrogram or linear filter-bank features,^[85] showing its superiority over the Mel-Cepstral features which contain a few stages of fixed transformation from spectrograms.
The true «raw» features of speech, waveforms, have more recently been shown to produce excellent larger-scale speech recognition results.^[86]

End-to-end automatic speech recognition[edit]

Since 2014, there has been much research interest in «end-to-end» ASR. Traditional phonetic-based (i.e., all HMM-based model) approaches required separate components and training for the pronunciation, acoustic, and language model. End-to-end models jointly learn all the components of the speech recognizer. This is valuable since it simplifies the training process and deployment process. For example, a n-gram language model is required for all HMM-based systems, and a typical n-gram language model often takes several gigabytes in memory making them impractical to deploy on mobile devices.^[87] Consequently, modern commercial ASR systems from Google and Apple (as of 2017) are deployed on the cloud and require a network connection as opposed to the device locally.

The first attempt at end-to-end ASR was with Connectionist Temporal Classification (CTC)-based systems introduced by Alex Graves of Google DeepMind and Navdeep Jaitly of the University of Toronto in 2014.^[88] The model consisted of recurrent neural networks and a CTC layer. Jointly, the RNN-CTC model learns the pronunciation and acoustic model together, however it is incapable of learning the language due to conditional independence assumptions similar to a HMM. Consequently, CTC models can directly learn to map speech acoustics to English characters, but the models make many common spelling mistakes and must rely on a separate language model to clean up the transcripts. Later, Baidu expanded on the work with extremely large datasets and demonstrated some commercial success in Chinese Mandarin and English.^[89] In 2016, University of Oxford presented LipNet,^[90] the first end-to-end sentence-level lipreading model, using spatiotemporal convolutions coupled with an RNN-CTC architecture, surpassing human-level performance in a restricted grammar dataset.^[91] A large-scale CNN-RNN-CTC architecture was presented in 2018 by Google DeepMind achieving 6 times better performance than human experts.^[92]

An alternative approach to CTC-based models are attention-based models. Attention-based ASR models were introduced simultaneously by Chan et al. of Carnegie Mellon University and Google Brain and Bahdanau et al. of the University of Montreal in 2016.^[93][94] The model named «Listen, Attend and Spell» (LAS), literally «listens» to the acoustic signal, pays «attention» to different parts of the signal and «spells» out the transcript one character at a time. Unlike CTC-based models, attention-based models do not have conditional-independence assumptions and can learn all the components of a speech recognizer including the pronunciation, acoustic and language model directly. This means, during deployment, there is no need to carry around a language model making it very practical for applications with limited memory. By the end of 2016, the attention-based models have seen considerable success including outperforming the CTC models (with or without an external language model).^[95] Various extensions have been proposed since the original LAS model. Latent Sequence Decompositions (LSD) was proposed by Carnegie Mellon University, MIT and Google Brain to directly emit sub-word units which are more natural than English characters;^[96] University of Oxford and Google DeepMind extended LAS to «Watch, Listen, Attend and Spell» (WLAS) to handle lip reading surpassing human-level performance.^[97]

Applications[edit]

In-car systems[edit]

Typically a manual control input, for example by means of a finger control on the steering-wheel, enables the speech recognition system and this is signaled to the driver by an audio prompt. Following the audio prompt, the system has a «listening window» during which it may accept a speech input for recognition.^{[citation needed]}

Simple voice commands may be used to initiate phone calls, select radio stations or play music from a compatible smartphone, MP3 player or music-loaded flash drive. Voice recognition capabilities vary between car make and model. Some of the most recent^[when?] car models offer natural-language speech recognition in place of a fixed set of commands, allowing the driver to use full sentences and common phrases. With such systems there is, therefore, no need for the user to memorize a set of fixed command words.^{[citation needed]}

Health care[edit]

Medical documentation[edit]

In the health care sector, speech recognition can be implemented in front-end or back-end of the medical documentation process. Front-end speech recognition is where the provider dictates into a speech-recognition engine, the recognized words are displayed as they are spoken, and the dictator is responsible for editing and signing off on the document. Back-end or deferred speech recognition is where the provider dictates into a digital dictation system, the voice is routed through a speech-recognition machine and the recognized draft document is routed along with the original voice file to the editor, where the draft is edited and report finalized. Deferred speech recognition is widely used in the industry currently.

One of the major issues relating to the use of speech recognition in healthcare is that the American Recovery and Reinvestment Act of 2009 (ARRA) provides for substantial financial benefits to physicians who utilize an EMR according to «Meaningful Use» standards. These standards require that a substantial amount of data be maintained by the EMR (now more commonly referred to as an Electronic Health Record or EHR). The use of speech recognition is more naturally suited to the generation of narrative text, as part of a radiology/pathology interpretation, progress note or discharge summary: the ergonomic gains of using speech recognition to enter structured discrete data (e.g., numeric values or codes from a list or a controlled vocabulary) are relatively minimal for people who are sighted and who can operate a keyboard and mouse.

A more significant issue is that most EHRs have not been expressly tailored to take advantage of voice-recognition capabilities. A large part of the clinician’s interaction with the EHR involves navigation through the user interface using menus, and tab/button clicks, and is heavily dependent on keyboard and mouse: voice-based navigation provides only modest ergonomic benefits. By contrast, many highly customized systems for radiology or pathology dictation implement voice «macros», where the use of certain phrases – e.g., «normal report», will automatically fill in a large number of default values and/or generate boilerplate, which will vary with the type of the exam – e.g., a chest X-ray vs. a gastrointestinal contrast series for a radiology system.

Therapeutic use[edit]

Prolonged use of speech recognition software in conjunction with word processors has shown benefits to short-term-memory restrengthening in brain AVM patients who have been treated with resection. Further research needs to be conducted to determine cognitive benefits for individuals whose AVMs have been treated using radiologic techniques.^{[citation needed]}

Military[edit]

High-performance fighter aircraft[edit]

Substantial efforts have been devoted in the last decade to the test and evaluation of speech recognition in fighter aircraft. Of particular note have been the US program in speech recognition for the Advanced Fighter Technology Integration (AFTI)/F-16 aircraft (F-16 VISTA), the program in France for Mirage aircraft, and other programs in the UK dealing with a variety of aircraft platforms. In these programs, speech recognizers have been operated successfully in fighter aircraft, with applications including setting radio frequencies, commanding an autopilot system, setting steer-point coordinates and weapons release parameters, and controlling flight display.

Working with Swedish pilots flying in the JAS-39 Gripen cockpit, Englund (2004) found recognition deteriorated with increasing g-loads. The report also concluded that adaptation greatly improved the results in all cases and that the introduction of models for breathing was shown to improve recognition scores significantly. Contrary to what might have been expected, no effects of the broken English of the speakers were found. It was evident that spontaneous speech caused problems for the recognizer, as might have been expected. A restricted vocabulary, and above all, a proper syntax, could thus be expected to improve recognition accuracy substantially.^[98]

The Eurofighter Typhoon, currently in service with the UK RAF, employs a speaker-dependent system, requiring each pilot to create a template. The system is not used for any safety-critical or weapon-critical tasks, such as weapon release or lowering of the undercarriage, but is used for a wide range of other cockpit functions. Voice commands are confirmed by visual and/or aural feedback. The system is seen as a major design feature in the reduction of pilot workload,^[99] and even allows the pilot to assign targets to his aircraft with two simple voice commands or to any of his wingmen with only five commands.^[100]

Speaker-independent systems are also being developed and are under test for the F35 Lightning II (JSF) and the Alenia Aermacchi M-346 Master lead-in fighter trainer. These systems have produced word accuracy scores in excess of 98%.^[101]

Helicopters[edit]

The problems of achieving high recognition accuracy under stress and noise are particularly relevant in the helicopter environment as well as in the jet fighter environment. The acoustic noise problem is actually more severe in the helicopter environment, not only because of the high noise levels but also because the helicopter pilot, in general, does not wear a facemask, which would reduce acoustic noise in the microphone. Substantial test and evaluation programs have been carried out in the past decade in speech recognition systems applications in helicopters, notably by the U.S. Army Avionics Research and Development Activity (AVRADA) and by the Royal Aerospace Establishment (RAE) in the UK. Work in France has included speech recognition in the Puma helicopter. There has also been much useful work in Canada. Results have been encouraging, and voice applications have included: control of communication radios, setting of navigation systems, and control of an automated target handover system.

As in fighter applications, the overriding issue for voice in helicopters is the impact on pilot effectiveness. Encouraging results are reported for the AVRADA tests, although these represent only a feasibility demonstration in a test environment. Much remains to be done both in speech recognition and in overall speech technology in order to consistently achieve performance improvements in operational settings.

Training air traffic controllers[edit]

Training for air traffic controllers (ATC) represents an excellent application for speech recognition systems. Many ATC training systems currently require a person to act as a «pseudo-pilot», engaging in a voice dialog with the trainee controller, which simulates the dialog that the controller would have to conduct with pilots in a real ATC situation. Speech recognition and synthesis techniques offer the potential to eliminate the need for a person to act as a pseudo-pilot, thus reducing training and support personnel. In theory, Air controller tasks are also characterized by highly structured speech as the primary output of the controller, hence reducing the difficulty of the speech recognition task should be possible. In practice, this is rarely the case. The FAA document 7110.65 details the phrases that should be used by air traffic controllers. While this document gives less than 150 examples of such phrases, the number of phrases supported by one of the simulation vendors speech recognition systems is in excess of 500,000.

The USAF, USMC, US Army, US Navy, and FAA as well as a number of international ATC training organizations such as the Royal Australian Air Force and Civil Aviation Authorities in Italy, Brazil, and Canada are currently using ATC simulators with speech recognition from a number of different vendors.^{[citation needed]}

Telephony and other domains[edit]

ASR is now commonplace in the field of telephony and is becoming more widespread in the field of computer gaming and simulation. In telephony systems, ASR is now being predominantly used in contact centers by integrating it with IVR systems. Despite the high level of integration with word processing in general personal computing, in the field of document production, ASR has not seen the expected increases in use.

The improvement of mobile processor speeds has made speech recognition practical in smartphones. Speech is used mostly as a part of a user interface, for creating predefined or custom speech commands.

Usage in education and daily life[edit]

For language learning, speech recognition can be useful for learning a second language. It can teach proper pronunciation, in addition to helping a person develop fluency with their speaking skills.^[102]

Students who are blind (see Blindness and education) or have very low vision can benefit from using the technology to convey words and then hear the computer recite them, as well as use a computer by commanding with their voice, instead of having to look at the screen and keyboard.^[103]

Students who are physically disabled , have a Repetitive strain injury/other injuries to the upper extremities can be relieved from having to worry about handwriting, typing, or working with scribe on school assignments by using speech-to-text programs. They can also utilize speech recognition technology to enjoy searching the Internet or using a computer at home without having to physically operate a mouse and keyboard.^[103]

Speech recognition can allow students with learning disabilities to become better writers. By saying the words aloud, they can increase the fluidity of their writing, and be alleviated of concerns regarding spelling, punctuation, and other mechanics of writing.^[104] Also, see Learning disability.

The use of voice recognition software, in conjunction with a digital audio recorder and a personal computer running word-processing software has proven to be positive for restoring damaged short-term memory capacity, in stroke and craniotomy individuals.

People with disabilities[edit]

People with disabilities can benefit from speech recognition programs. For individuals that are Deaf or Hard of Hearing, speech recognition software is used to automatically generate a closed-captioning of conversations such as discussions in conference rooms, classroom lectures, and/or religious services.^[105]

Speech recognition is also very useful for people who have difficulty using their hands, ranging from mild repetitive stress injuries to involve disabilities that preclude using conventional computer input devices. In fact, people who used the keyboard a lot and developed RSI became an urgent early market for speech recognition.^[106][107] Speech recognition is used in deaf telephony, such as voicemail to text, relay services, and captioned telephone. Individuals with learning disabilities who have problems with thought-to-paper communication (essentially they think of an idea but it is processed incorrectly causing it to end up differently on paper) can possibly benefit from the software but the technology is not bug proof.^[108] Also the whole idea of speak to text can be hard for intellectually disabled person’s due to the fact that it is rare that anyone tries to learn the technology to teach the person with the disability.^[109]

This type of technology can help those with dyslexia but other disabilities are still in question. The effectiveness of the product is the problem that is hindering it from being effective. Although a kid may be able to say a word depending on how clear they say it the technology may think they are saying another word and input the wrong one. Giving them more work to fix, causing them to have to take more time with fixing the wrong word.^[110]

Further applications[edit]

• Aerospace (e.g. space exploration, spacecraft, etc.) NASA’s Mars Polar Lander used speech recognition technology from Sensory, Inc. in the Mars Microphone on the Lander^[111]
• Automatic subtitling with speech recognition
• Automatic emotion recognition^[112]
• Automatic shot listing in audiovisual production
• Automatic translation
• eDiscovery (Legal discovery)
• Hands-free computing: Speech recognition computer user interface
• Home automation
• Interactive voice response
• Mobile telephony, including mobile email
• Multimodal interaction^[62]
• Pronunciation evaluation in computer-aided language learning applications
• Real Time Captioning^[113]
• Robotics
• Security, including usage with other biometric scanners for multi-factor authentication^[114]
• Speech to text (transcription of speech into text, real time video captioning, Court reporting )
• Telematics (e.g. vehicle Navigation Systems)
• Transcription (digital speech-to-text)
• Video games, with Tom Clancy’s EndWar and Lifeline as working examples
• Virtual assistant (e.g. Apple’s Siri)

Performance[edit]

The performance of speech recognition systems is usually evaluated in terms of accuracy and speed.^[115][116] Accuracy is usually rated with word error rate (WER), whereas speed is measured with the real time factor. Other measures of accuracy include Single Word Error Rate (SWER) and Command Success Rate (CSR).

Speech recognition by machine is a very complex problem, however. Vocalizations vary in terms of accent, pronunciation, articulation, roughness, nasality, pitch, volume, and speed. Speech is distorted by a background noise and echoes, electrical characteristics. Accuracy of speech recognition may vary with the following:^{[117][citation needed]}

• Vocabulary size and confusability
• Speaker dependence versus independence
• Isolated, discontinuous or continuous speech
• Task and language constraints
• Read versus spontaneous speech
• Adverse conditions

Accuracy[edit]

As mentioned earlier in this article, the accuracy of speech recognition may vary depending on the following factors:

• Error rates increase as the vocabulary size grows:

e.g. the 10 digits «zero» to «nine» can be recognized essentially perfectly, but vocabulary sizes of 200, 5000 or 100000 may have error rates of 3%, 7%, or 45% respectively.

• Vocabulary is hard to recognize if it contains confusing words:

e.g. the 26 letters of the English alphabet are difficult to discriminate because they are confusing words (most notoriously, the E-set: «B, C, D, E, G, P, T, V, Z — when «Z» is pronounced «zee» rather than «zed» depending on the English region); an 8% error rate is considered good for this vocabulary.^{[citation needed]}

• Speaker dependence vs. independence:

A speaker-dependent system is intended for use by a single speaker.

A speaker-independent system is intended for use by any speaker (more difficult).

• Isolated, Discontinuous or continuous speech

With isolated speech, single words are used, therefore it becomes easier to recognize the speech.

With discontinuous speech full sentences separated by silence are used, therefore it becomes easier to recognize the speech as well as with isolated speech.
With continuous speech naturally spoken sentences are used, therefore it becomes harder to recognize the speech, different from both isolated and discontinuous speech.

• Task and language constraints
  • e.g. Querying application may dismiss the hypothesis «The apple is red.»
  • e.g. Constraints may be semantic; rejecting «The apple is angry.»
  • e.g. Syntactic; rejecting «Red is apple the.»

Constraints are often represented by grammar.

• Read vs. Spontaneous Speech – When a person reads it’s usually in a context that has been previously prepared, but when a person uses spontaneous speech, it is difficult to recognize the speech because of the disfluencies (like «uh» and «um», false starts, incomplete sentences, stuttering, coughing, and laughter) and limited vocabulary.
• Adverse conditions – Environmental noise (e.g. Noise in a car or a factory). Acoustical distortions (e.g. echoes, room acoustics)

Speech recognition is a multi-leveled pattern recognition task.

• Acoustical signals are structured into a hierarchy of units, e.g. Phonemes, Words, Phrases, and Sentences;
• Each level provides additional constraints;

e.g. Known word pronunciations or legal word sequences, which can compensate for errors or uncertainties at a lower level;

• This hierarchy of constraints is exploited. By combining decisions probabilistically at all lower levels, and making more deterministic decisions only at the highest level, speech recognition by a machine is a process broken into several phases. Computationally, it is a problem in which a sound pattern has to be recognized or classified into a category that represents a meaning to a human. Every acoustic signal can be broken into smaller more basic sub-signals. As the more complex sound signal is broken into the smaller sub-sounds, different levels are created, where at the top level we have complex sounds, which are made of simpler sounds on the lower level, and going to lower levels, even more, we create more basic and shorter and simpler sounds. At the lowest level, where the sounds are the most fundamental, a machine would check for simple and more probabilistic rules of what sound should represent. Once these sounds are put together into more complex sounds on upper level, a new set of more deterministic rules should predict what the new complex sound should represent. The most upper level of a deterministic rule should figure out the meaning of complex expressions. In order to expand our knowledge about speech recognition, we need to take into consideration neural networks. There are four steps of neural network approaches:
• Digitize the speech that we want to recognize

For telephone speech the sampling rate is 8000 samples per second;

• Compute features of spectral-domain of the speech (with Fourier transform);

computed every 10 ms, with one 10 ms section called a frame;

Analysis of four-step neural network approaches can be explained by further information. Sound is produced by air (or some other medium) vibration, which we register by ears, but machines by receivers. Basic sound creates a wave which has two descriptions: amplitude (how strong is it), and frequency (how often it vibrates per second).
Accuracy can be computed with the help of word error rate (WER). Word error rate can be calculated by aligning the recognized word and referenced word using dynamic string alignment. The problem may occur while computing the word error rate due to the difference between the sequence lengths of the recognized word and referenced word.

The formula to compute the word error rate (WER) is:

where s is the number of substitutions, d is the number of deletions, i is the number of insertions, and n is the number of word references.

While computing, the word recognition rate (WRR) is used. The formula is:

where h is the number of correctly recognized words:

.

Security concerns[edit]

Speech recognition can become a means of attack, theft, or accidental operation. For example, activation words like «Alexa» spoken in an audio or video broadcast can cause devices in homes and offices to start listening for input inappropriately, or possibly take an unwanted action.^[118] Voice-controlled devices are also accessible to visitors to the building, or even those outside the building if they can be heard inside. Attackers may be able to gain access to personal information, like calendar, address book contents, private messages, and documents. They may also be able to impersonate the user to send messages or make online purchases.

Two attacks have been demonstrated that use artificial sounds. One transmits ultrasound and attempt to send commands without nearby people noticing.^[119] The other adds small, inaudible distortions to other speech or music that are specially crafted to confuse the specific speech recognition system into recognizing music as speech, or to make what sounds like one command to a human sound like a different command to the system.^[120]

Further information[edit]

Conferences and journals[edit]

Popular speech recognition conferences held each year or two include SpeechTEK and SpeechTEK Europe, ICASSP, Interspeech/Eurospeech, and the IEEE ASRU. Conferences in the field of natural language processing, such as ACL, NAACL, EMNLP, and HLT, are beginning to include papers on speech processing. Important journals include the IEEE Transactions on Speech and Audio Processing (later renamed IEEE Transactions on Audio, Speech and Language Processing and since Sept 2014 renamed IEEE/ACM Transactions on Audio, Speech and Language Processing—after merging with an ACM publication), Computer Speech and Language, and Speech Communication.

Books[edit]

Books like «Fundamentals of Speech Recognition» by Lawrence Rabiner can be useful to acquire basic knowledge but may not be fully up to date (1993). Another good source can be «Statistical Methods for Speech Recognition» by Frederick Jelinek and «Spoken Language Processing (2001)» by Xuedong Huang etc., «Computer Speech», by Manfred R. Schroeder, second edition published in 2004, and «Speech Processing: A Dynamic and Optimization-Oriented Approach» published in 2003 by Li Deng and Doug O’Shaughnessey. The updated textbook Speech and Language Processing (2008) by Jurafsky and Martin presents the basics and the state of the art for ASR. Speaker recognition also uses the same features, most of the same front-end processing, and classification techniques as is done in speech recognition. A comprehensive textbook, «Fundamentals of Speaker Recognition» is an in depth source for up to date details on the theory and practice.^[121] A good insight into the techniques used in the best modern systems can be gained by paying attention to government sponsored evaluations such as those organised by DARPA (the largest speech recognition-related project ongoing as of 2007 is the GALE project, which involves both speech recognition and translation components).

A good and accessible introduction to speech recognition technology and its history is provided by the general audience book «The Voice in the Machine. Building Computers That Understand Speech» by Roberto Pieraccini (2012).

The most recent book on speech recognition is Automatic Speech Recognition: A Deep Learning Approach (Publisher: Springer) written by Microsoft researchers D. Yu and L. Deng and published near the end of 2014, with highly mathematically oriented technical detail on how deep learning methods are derived and implemented in modern speech recognition systems based on DNNs and related deep learning methods.^[82] A related book, published earlier in 2014, «Deep Learning: Methods and Applications» by L. Deng and D. Yu provides a less technical but more methodology-focused overview of DNN-based speech recognition during 2009–2014, placed within the more general context of deep learning applications including not only speech recognition but also image recognition, natural language processing, information retrieval, multimodal processing, and multitask learning.^[78]

Software[edit]

In terms of freely available resources, Carnegie Mellon University’s Sphinx toolkit is one place to start to both learn about speech recognition and to start experimenting. Another resource (free but copyrighted) is the HTK book (and the accompanying HTK toolkit). For more recent and state-of-the-art techniques, Kaldi toolkit can be used.^[122] In 2017 Mozilla launched the open source project called Common Voice^[123] to gather big database of voices that would help build free speech recognition project DeepSpeech (available free at GitHub),^[124] using Google’s open source platform TensorFlow.^[125] When Mozilla redirected funding away from the project in 2020, it was forked by its original developers as Coqui STT^[126] using the same open-source license.^[127][128]

Google Gboard supports speech recognition on all Android applications. It can be activated through the microphone icon.^[129]

The commercial cloud based speech recognition APIs are broadly available.

For more software resources, see List of speech recognition software.

See also[edit]

References[edit]

Further reading[edit]

• Pieraccini, Roberto (2012). The Voice in the Machine. Building Computers That Understand Speech. The MIT Press. ISBN 978-0262016858.
• Woelfel, Matthias; McDonough, John (26 May 2009). Distant Speech Recognition. Wiley. ISBN 978-0470517048.
• Karat, Clare-Marie; Vergo, John; Nahamoo, David (2007). «Conversational Interface Technologies». In Sears, Andrew; Jacko, Julie A. (eds.). The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies, and Emerging Applications (Human Factors and Ergonomics). Lawrence Erlbaum Associates Inc. ISBN 978-0-8058-5870-9.
• Cole, Ronald; Mariani, Joseph; Uszkoreit, Hans; Varile, Giovanni Battista; Zaenen, Annie; Zampolli; Zue, Victor, eds. (1997). Survey of the state of the art in human language technology. Cambridge Studies in Natural Language Processing. Vol. XII–XIII. Cambridge University Press. ISBN 978-0-521-59277-2.
• Junqua, J.-C.; Haton, J.-P. (1995). Robustness in Automatic Speech Recognition: Fundamentals and Applications. Kluwer Academic Publishers. ISBN 978-0-7923-9646-8.
• Pirani, Giancarlo, ed. (2013). Advanced algorithms and architectures for speech understanding. Springer Science & Business Media. ISBN 978-3-642-84341-9.

External links[edit]

• Signer, Beat and Hoste, Lode: SpeeG2: A Speech- and Gesture-based Interface for Efficient Controller-free Text Entry, In Proceedings of ICMI 2013, 15th International Conference on Multimodal Interaction, Sydney, Australia, December 2013
• Speech Technology at Curlie

People recognize and distinguish each other’s voices almost immediately. But what comes naturally for a human is challenging for a machine learning (ML) system. To make your speaker recognition solution efficient and performant, you need to carefully choose a model and train it on the most fitting dataset with the right parameters.

In this article, we briefly overview the key speaker recognition approaches along with tools, techniques, and models you can use for building a speaker recognition system. We also analyze and compare the performance of these models when configured with different parameters and trained with different datasets. This overview will be useful for teams working on speech processing and speaker recognition projects.

Understanding the basics of speaker recognition

Like a person’s retina and fingerprints, a person’s voice is a unique identifier. That’s why speaker recognition is widely applied for building human-to-machine interaction and biometric solutions like voice assistants, voice-controlled services, and speech-based authentication products.

To provide personalized services and correctly authenticate users, such systems should be able to recognize a user. To do so, modern speech processing solutions often rely on speaker recognition.

Speaker recognition verifies a person’s identity (or identifies a person) by analyzing voice characteristics.

There are two types of speaker recognition:

• Speaker identification — The system identifies a speaker by comparing their speech to models of known speakers.
• Speaker verification — The system verifies that the speaker is who they claim to be.

You can build a speaker recognition system using static signal processing, machine learning algorithms, neural networks, and other technologies. In this article, we focus on the specifics of accomplishing speaker recognition tasks using machine learning algorithms.

Both speaker verification and speaker identification systems consist of three main stages:

• Feature extraction is when the system extracts essential voice features from raw audio
• Speaker modeling is when the system creates a probabilistic model for each speaker
• Matching is when the system compares the input voice with every previously created model and chooses the model that best matches the input voice

Implementing each of these stages requires different tools and techniques. We look closely at some of them in the next section.

Speech is the key element in speaker recognition. And to work with speech, you’ll need to reduce noise, distinguish parts of speech from silence, and extract particular speech features. But first, you’ll need to properly prepare your speech recordings for further processing.

Speech signal preprocessing

Converting speech audio to the data format used by the ML system is the initial step of the speaker recognition process.

Start by recording speech with a microphone and turning the audio signal into digital data with an analog-to-digital converter. Further signal processing commonly includes processes like voice activity detection (VAD), noise reduction, and feature extraction. We’ll look at each of these processes later.

First, let’s overview some of the key speech signal preprocessing techniques: feature scaling and stereo-to-mono conversion.

Since the range of signal values varies widely, some machine learning algorithms can’t properly recognize audio without normalization. Feature scaling is a method used to normalize the range of independent variables or features of data. Scaling data eliminates sparsity by bringing all your values onto the same scale, following the same concept as normalization and standardization.

For example, you can standardize your audio data using the sklearn.preprocessing package. It contains utility functions and transformer classes that allow you to improve the representation of raw feature vectors.

Here’s how this works in practice:

from sklearn import preprocessing
def extract_features(audio,rate):
   mfcc_feature = mfcc.mfcc(audio,rate, winlen=0.020,preemph=0.95,numcep=20,nfft=1024,ceplifter=15,highfreq=6000,nfilt=55,appendEnergy=False)
   mfcc_feature = preprocessing.scale(mfcc_feature)
   delta = calculate_delta(mfcc_feature)
   combined = np.hstack((mfcc_feature,delta))
   return combined

The number of channels in an audio file can also influence the performance of your speaker recognition system. Audio files can be recorded in mono or stereo format: mono audio has only one channel, while stereo audio has two or more channels. Converting stereo recordings to mono helps to improve the accuracy and performance of a speaker recognition system.

Python provides a pydub module that enables you to play, split, merge, and edit WAV audio files. This is how you can use it to convert a stereo WAV file to a mono file:

from pydub import AudioSegment
mysound = AudioSegment.from_wav("stereo_infile.wav")
# set mono channel
mysound = mysound.set_channels(1)
# save the result
mysound.export("mono_outfile.wav", format="wav")

Now that we’ve overviewed key preparation measures, we can dive deeper into the specifics of speech signal processing techniques.

Voice activity detection

To let the feature extraction algorithm for speaker recognition focus only on speech, we can remove silence and non-speech parts of the audio. This is where VAD comes in handy. This technique can distinguish human speech from other signals and is often used in speech-controlled applications and devices like voice assistants and smartphones.

The main goal of a voice activity detection algorithm is to determine which segments of a signal are speech and which are not. A VAD algorithm can improve the performance of a speaker verification system by making sure that the speaker’s identity is calculated using only the frames that contain speech. Therefore, you should evaluate the necessity of using a VAD algorithm to overcome problems in designing a robust speaker verification system.

As for the practical aspects of VAD implementation, you can turn your attention to PyTorch, a high-performance open-source library with a rich variety of deep learning (DL) algorithms.

Here’s how you can use PyTorch to detect voice activity in a recording:

import torch


# loading vad model and tools to work with audio
model, utils = torch.hub.load(repo_or_dir='snakers4/silero-vad', model='silero_vad', force_reload=False)

(_, get_speech_ts_adaptive, save_audio, read_audio, _, _, collect_chunks) = utils

audio = read_audio('raw_voice.wav')


# get time chunks with voice
speech_timestamps = get_speech_ts_adaptive(wav, model)


# gather the chunks and save them to a file
save_audio('only_speech.wav',
       	collect_chunks(speech_timestamps, audio))

Noise reduction

Noise is inevitably present in almost all acoustic environments. Even when recorded with a microphone, a speech signal will contain lots of noise, such as white noise or background sounds.

Excessive noise can distort or mask the characteristics of speech signals, degrading the overall quality of the speech recording. The more noise an audio signal contains, the poorer the performance of a human-to-machine communication system.

Noise detection and reduction are often formulated as a digital filtering problem, where you get clean speech by passing noisy speech through a linear filter. In this case, the key challenge is to design a filter that can significantly suppress noise without causing any noticeable speech distortion.

Developing a versatile noise detection and reduction algorithm that works in diverse environments is a challenging task, as noise characteristics are inconsistent.

You can use these two tools to successfully handle noise detection and reduction tasks:

• Noisereduce is a Python noise reduction algorithm that you can use to reduce the level of noise in speech and time-domain signals. It includes two algorithms for stationary and non-stationary noise reduction.
• SciPy is an open-source collection of mathematical algorithms that you can use to manipulate and visualize data using high-level Python commands.

Here’s an example of working with SciPy:

import noisereduce as nr
from scipy.io import wavfile

# load data
rate, data = wavfile.read("voice_with_noise.wav")

# perform noise reduction
reduced_noise = nr.reduce_noise(y=data, sr=rate)

Feature extraction

Feature extraction is the process of identifying unique characteristics in a speech signal, transforming raw acoustic signals into a compact representation. There are various techniques to extract features from speech samples: Linear Predictive Coding, Mel Frequency Cepstral Coefficient (MFCC), Power Normalized Cepstral Coefficients, and Gammatone Frequency Cepstral Coefficients, to name a few.

In this section, we’ll focus on two popular feature extraction techniques:

• Mel-frequency cepstrum (MFCC)
• Delta MFCC

MFCC is a feature extractor commonly used for speech recognition. It works somewhat similarly to the human ear, representing sound in both linear and non-linear cepstrals.

If we take the first derivative of an MFCC feature, we can extract a Delta MFCC feature from it. In contrast to general MFCC features, Delta MFCC features can be used to represent temporal information. In particular, you can use these features to represent changes between frames.

You can find these three tools helpful when working on feature extraction in a speaker recognition system:

• NumPy is an open-source Python module providing you with a high-performance multidimensional array object and a wide selection of functions for working with arrays.
• Scikit-learn is a free ML library for Python that features different classification, regression, and clustering algorithms. You can use Scikit-learn along with the NumPy and SciPy libraries.
• Python_speech_features is another Python library that you can use for working with MFCCs.

Here’s an example of using delta MFCC and combining it with a regular MFCC:

import numpy as np
from sklearn import preprocessing
from python_speech_features import mfcc, delta

def extract_features(audio,rate):
	"""extract 20 dim mfcc features from audio file, perform CMS and combine
	delta to make 40 dim feature vector"""    
   
	mfcc_feature = mfcc.mfcc(audio, rate, winlen=0.020,preemph=0.95,numcep=20,nfft=1024,ceplifter=15,highfreq=6000,nfilt=55,appendEnergy=False)
 
	# feature scaling
	mfcc_feature = preprocessing.scale(mfcc_feature)
	delta_feature = delta(mfcc_feature, 2) # calculating delta
	# stacking delta features with common features
	combined_features = np.hstack((mfcc_feature, delta_feature))
	return combined_features

Now that you know what speaker recognition techniques and tools you can use, it’s time to see the ML and DL models and algorithms that can help you build an efficient speaker recognition system. You can also use these techniques to figure out how to make an IVR system for real time voice responce.

ML and DL models for speaker recognition

While you can always try building a custom machine learning model from scratch, using an already trained and tested algorithm or model can save both time and money for your speaker recognition project. Below, we take a look at five ML and DL models commonly applied for speech processing and speaker recognition tasks.

We’ll start with one of the most popular models for processing audio data — the Gaussian Mixture Model.

Gaussian Mixture Model

The Gaussian Mixture Model (GMM) is an unsupervised machine learning model commonly used for solving data clustering and data mining tasks. This model relies on Gaussian distributions, assuming there is a certain number of them, each representing a separate cluster. GMMs tend to group data points from a single distribution together.

Combining a GMM with the MFCC feature extraction technique provides great accuracy when completing speaker recognition tasks. The GMM is trained using the expectation maximization algorithm, which creates gaussian mixtures by updating gaussian means based on the maximum likelihood estimate.

To work with GMM algorithms, you can use the sklearn.mixture package, which helps you learn from and sample different GMMs.

Here’s how you extract features (using the previously described method) and train the GMM using sklearn.mixture:

sample_rate, data = read('denoised_vad_voice.wav')

# extract 40 dimensional MFCC & delta MFCC features
features = extract_features(audio, sr)

gmm = GMM(n_components=16,max_iter=200,covariance_type='diag',n_init=1, init_params='random')
gmm.fit(features)  # gmm training

Combining the Gaussian Mixture Model and Universal Background Model

A GMM is usually trained on speech samples from a particular speaker, distinguishing speech features unique to that person.

When trained on a large set of speech samples, a GMM can learn general speech characteristics and turn into a Universal Background Model (UBM). UBMs are commonly used in biometric verification solutions to represent person-independent speech features.

Combined in a single system, GMM and UBM models can better handle alternative speech they may encounter during speaker recognition, like whispers, slow speech, or fast speech. This applies to both the type and the quality of speech as well as the composition of speakers.

Note that in typical speaker verification tasks, a speaker model can’t be identified directly with the help of the expectation maximization algorithm, as the variety of training data is limited. For this reason, speaker models in speaker verification systems are often trained using the maximum a posteriori (MAP) adaptation, which can estimate a speaker model from UBM.

As for tools, you can use Kaldi — a popular speech recognition toolset for clustering and feature extraction. To integrate its functionality with Python-based workflows, you can go with the Bob.kaldi package that provides pythonic bindings for Kaldi.

Here’s an example of working with Bob.kaldi:

ubm = bob.kaldi.ubm_train(features, r'ubm_vad.h5', num_threads=4, num_gauss=2048, num_iters=100)


# training every gmm using ubm
user_model = bob.kaldi.ubm_enroll(features, ubm)


# scoring using ubm and a specified gmm
bob.kaldi.gmm_score(features, user_model, ubm)

To achieve even higher accuracy, you can try accomplishing speaker recognition tasks with deep learning algorithms. Let’s look at some of the commonly applied DL models in the next section.

Deep learning models for speaker recognition

When trying to solve speaker recognition problems with deep learning algorithms, you’ll probably need to use a convolutional neural network (CNN). While this type of neural network is widely applied for solving image-related problems, some models were designed specifically for speech processing:

VGGVox is a DL system that can be used for both speaker verification and speaker identification. The network architecture allows you to extract frame-level features from the spectrograms and aggregate them to obtain an audio voice print. To verify a speaker, the system compares voice prints using cosine distance — for similar voice prints, this distance will always be rather small.

To train this model, you need to preprocess your audio data by converting regular audio to the mono format and generating spectrograms out of it. Then you can feed normalized spectrograms to the CNN model in the form of images.

Deep speaker is a Residual CNN–based model for speech processing and recognition. After passing speech features through the network, we get speaker embeddings that are additionally normalized in order to have unit norm.

Just like VGGVox, the system checks the similarity of speakers using the cosine distance method. But unlike VGGVox, Deep speaker computes loss using the triplet loss method. The main idea of this method is to maximize the cosine similarities of two different embeddings of the same speaker while minimizing embeddings of another speaker.

Audio preprocessing for this system includes converting your audio files to 64-dimensional filter bank coefficients and normalizing the results so they have zero mean and unit variance.

SpeakerNet is a neural network that can be used for both speaker verification and speaker identification and can easily be integrated with other deep automatic speech recognition models.

The preprocessing stage for this model also includes generating spectrograms from raw audio and feeding them to the neural network as images.

When working with these models, you can rely on the glob module that helps you find all path names matching a particular pattern. You can also use the malaya_speech library that provides a rich selection of speech processing features.

Here’s what working with these DL models can look like in practice:

# load model
def deep_model(model: str = 'speakernet', quantized: bool = False, **kwargs):
	"""
	Load Speaker2Vec model.

	Parameters
	----------
	model : str, optional (default='speakernet')
    	Model architecture supported. Allowed values:

    	* ``'vggvox-v1'`` - VGGVox V1, embedding size 1024
    	* ``'vggvox-v2'`` - VGGVox V2, embedding size 512
    	* ``'deep-speaker'`` - Deep Speaker, embedding size 512
    	* ``'speakernet'`` - SpeakerNet, embedding size 7205

	quantized : bool, optional (default=False)
    	if True, will load 8-bit quantized model.
    	The quantized model isn’t necessarily faster, it totally depends on the machine.

	Returns
	-------
	result : malaya_speech.supervised.classification.load function
	"""

model = malaya_speech.speaker_vector.deep_model('speakernet')

from glob import glob

speakers = ['voice_0.wav', 'vocie_1.wav', 'voice_2.wav']

# pipeline
def load_wav(file):
	return malaya_speech.load(file)[0]

p = Pipeline()
frame = p.foreach_map(load_wav).map(model)
r = p.emit(speakers)

# calculate similarity
from scipy.spatial.distance import cdist

1 - cdist(r['speaker-vector'], r['speaker-vector'], metric = 'cosine')

However, choosing the right ML or DL model is not enough for building a well-performing speaker recognition system. In the next section, we discuss how setting the right parameters can greatly improve the performance of your machine learning models.

Improving model performance with parameter tuning

There are factors affecting speaker recognition accuracy and the performance of your model. One of these factors is hyperparameters — parameters that a machine learning model can’t learn directly within estimators. These parameters are usually specified manually when developing the ML model.

Setting the appropriate values for hyperparameters can significantly improve your model’s performance. Let’s take a look at ways you can choose the right parameters and cross-validate your model’s performance based on the example of the Scikit-learn library.

Choosing parameters with grid search

Note: To follow through with our examples, you need to install the Scikit-learn library.

There’s no way to know the best values for hyperparameters in advance. Therefore, you need to try all possible combinations to find the optimal values.

One way to train an ML model with different parameters and determine parameters with the best score is by using grid search.

Grid search is implemented using GridSearchCV, available in Scikit-learn’s model_selection package. In this process, the model only uses the parameters specified in the param_grid parameter. GridSearchCV can help you loop through the predefined hyperparameters and fit your estimator to your training set. Once you tune all the parameters, you’ll get a list of the best-fitting hyperparameters.

Along with performing grid search, GridSearchCV can perform cross-validation — the process of choosing the best-performing parameters by dividing the training and testing data in different ways.

For example, we can choose an 80/20 data splitting coefficient, meaning we’ll use 80% of data from a chosen dataset for training the model and the remaining 20% for testing the model. Once we decide on the coefficient, the cross-validation technique applies a specified number of combinations to this data to find out the best split.

This is where the pipeline comes into play.

Cross-validating a model with pipelines

A pipeline is used to assemble several steps that can be cross-validated while setting different parameters for a model.

There are must-have methods for pipelines:

• • The fit method allows you to learn from data
  • The transform or predict method processes the data and generates a prediction

Scikit-learn’s pipeline class is useful for encapsulating multiple transformers alongside an estimator into one object so you need to call critical methods like fit and predict only once.

We can get the pipeline class from the sklearn.pipeline module.

You can chain transformers and estimators into a sequence that functions as a single cohesive unit using Scikit-learn’s pipeline constructor. For example, if your model involves feature selection, standardization, and regression, you can encapsulate three corresponding classes using the pipeline module.

And if you are using custom functions to transform data at the preprocessing and feature extraction stages, you might need to write custom transfer and classifier classes to incorporate these actions into your pipeline.

Here’s an example of a custom transformer class:

import numpy as np
import warnings
from python_speech_features import mfcc, delta
from sklearn import preprocessing
from sklearn.utils.validation import check_is_fitted
 
warnings.filterwarnings('ignore')
from sklearn.base import BaseEstimator, TransformerMixin

And this is what a CustomTransformer(TransformerMixin, BaseEstimator) class would look like:

def __init__(self, winlen=0.020,preemph=0.95,numcep=20,nfft=1024,ceplifter=15,highfreq=6000,nfilt=55,appendEnergy=False):
    	self.winlen = winlen
    	self.preemph = preemph
    	self.numcep = numcep
    	self.nfft = nfft
    	self.ceplifter = ceplifter
    	self.highfreq = highfreq
    	self.nfilt = nfilt
    	self.appendEnergy = appendEnergy
 
	def transform(self, x):
    	""" A reference implementation of a transform function.
            	Parameters
            	----------
            	x : {array-like, sparse-matrix}, shape (n_samples, n_features)
                	The input samples.
            	Returns
            	-------
            	X_transformed : array, shape (n_samples, n_features)
                	The array containing the element-wise square roots of the values
                	in ``X``.
            	"""
    	# Check is fit has been called
    	check_is_fitted(self, 'n_features_')
 
    	# Check that the input is of the same shape as the one passed
    	# during fit.
    	if x.shape[1] != self.n_features_:
        	raise ValueError('Shape of input is different from what was seen'
                         	'in `fit`')
    	return self.signal_to_mfcc(x)
 
	def fit(self, x, y=None):
    	"""A reference implementation of a fitting function for a transformer.
            	Parameters
            	----------
            	x : {array-like, sparse matrix}, shape (n_samples, n_features)
                	The training input samples.
            	y : None
                	There is no need of a target in a transformer, yet the pipeline API
                	requires this parameter.
            	Returns
            	-------
            	self : object
                	Returns self.
            	"""
    	self.n_features_ = x.shape[1]
    	return self

If you need a custom classifier class, this is what it might look like:

import numpy as np
import warnings
from python_speech_features import mfcc, delta
from sklearn import preprocessing
from sklearn.utils.validation import check_is_fitted

warnings.filterwarnings('ignore')
from sklearn.base import BaseEstimator, TransformerMixin

You might also need to create additional methods to load data and extract features and sample rates.

Here’s how you can create a load data method:

def load_data(folder):
	x, y = [], []

	for voice_sample in list(glob.glob(rf'./{folder}/id*/id*')):
    	voice_sample_file_name = os.path.basename(voice_sample)
    	voice_class, _ = voice_sample_file_name.split("_")

    	features = read_wav(voice_sample)

    	x.append(features)
    	y.append(voice_class)

	return np.array(x, dtype=tuple), np.array(y)

And here’s what the code of a method to extract features and sample rate can look like:

def read_wav(fname):
	fs, signal = wavfile.read(fname)
	if len(signal.shape) != 1:
    	signal = signal[:,0]
	return fs, signal

The parameters for the param_grid module should be specified using documentation for the defined method. It’s important to mention the list of different values or the range of values.

If you’re using binary outcomes (true or false), you need to define only two values. When working with categorical values, you need to create a list of all possible string values.

Here’s an example of how to determine the best-fitting parameters using grid search and a pipeline:

x, y = load_data('test')


# creating pipeline of transformer and classifier
pipe = Pipeline([('scaler', CustomTransformer()), ('svc', CustomClassifier())])


# creating the parameters grid
param_grid = {
    'scaler__appendEnergy': [True, False],
    'scaler__winlen': [0.020, 0.025, 0.015, 0.01],
    'scaler__preemph': [0.95, 0.97, 1, 0.90, 0.5, 0.1],
    'scaler__numcep': [20, 13, 16],
    'scaler__nfft': [1024, 1200, 512],
    'scaler__ceplifter': [15, 22, 0],
    'scaler__highfreq': [6000],
    'scaler__nfilt':[55, 0, 22],
    'svc__n_components': [2 * i for i in range(0, 12, 1)],
    'svc__max_iter': list(range(50, 400, 50)),
    'svc__covariance_type': ['full', 'tied', 'diag', 'spherical'],
    'svc__n_init': list(range(1, 4, 1)),
    'svc__init_params': ['kmeans', 'random']
}
search = GridSearchCV(pipe, param_grid, n_jobs=-1)
# searching for appropriate parameters
search.fit(x, y)

To illustrate how different parameters can affect the performance of an ML model, we tested the previously discussed models on different datasets. Go to the next section to see the results of these tests.

Evaluating the impact of parameter tuning

We tested the performance of two machine learning models: a combination of GMM and MFCC and the GMM-UBM model. For better test result accuracy, we compared the performance of these models on two datasets:

1. LibriSpeech — This dataset is a collection of around 1,000 hours of audiobook recordings. The training data is split into three sets: two containing “clean” speech (100 hours and 360 hours) and one containing 500 hours of “other” speech, which is considered more challenging for an ML model to process. The test data is also split into two categories: clean and other.

Here’s the structure of the LibriSpeech dataset:

.- train-clean-100/
            	|
            	.- 19/
                	|
                	.- 198/
                	| |
                	| .- 19-198.trans.txt
                	| |   
                	| .- 19-198-0001.flac
                	| |
                	| .- 14-208-0002.flac
                	| |
                	| ...
                	|
                	.- 227/
                     	| ...

2. VoxCeleb1 — This is one of two audio-visual VoxCeleb datasets formed from YouTube interviews with celebrities. In contrast to LibriSpeech, this dataset doesn’t have many clean speech samples, as most interviews were recorded in noisy environments.

The dataset includes recordings of 1251 celebrities, with a separate file for each person. Each file has subfolders with WAV audio files, and each subfolder depicts the YouTube video that was used to create the speech samples.

Here’s the structure of this dataset:

.- vox1-dev-wav/
|        	|
|        	.- wav/
|            	|
|            	.- id10001/
|            	| 	|
|            	| 	.- 1zcIwhmdeo4/
|            	| 	|    	|
|            	| 	...  	.- 00001.wav
|            	|
|            	.- id10002/
|                  	|
|                  	...
|

First, we tested a simple GMM-MFCC model trained on the LibriSpeech dataset. The results are the following:

Table 1 – Testing a GMM-MFCC model on the LibriSpeech dataset

As this dataset contains clean speech samples, the results for LibriSpeech are always good, whether we use a GMM-MFCC, GMM-UBM, or any other machine learning model. For this reason, all other tests will be focused on the VoxCeleb dataset, which is a lot more challenging for a machine learning model to process.

Let’s start with the results of testing a GMM-MFCC model using the VoxCeleb dataset:

Table 2 – Testing a GMM-MFCC model on the VoxCeleb dataset

As you can see, the accuracy of our model decreased significantly, especially when applied to a larger number of users.

Now let’s take a look at the results of testing a GMM-UBM model trained on the VoxCeleb dataset:

Table 3 – Testing a GMM-UBM model on the VoxCeleb dataset

This model proved to be more accurate than the GMM-MFCC model.

We can also see how changing the parameters of a model affects its performance. For this example, we tested the MFCC model and its delta and delta + double delta versions on 100 users:

Table 4 – Testing the influence of parameter changes

The numcep value represents the number of cepstrums to return. As you can see, the higher the numcep, the more accurate the results our models show.

And as for the deep learning models mentioned in this article, they showed the following results when trained on the VoxCeleb dataset and tested on 100 users:

Table 5 – Testing DL models on the VoxCeleb dataset

As you can see, the performance of different ML and DL methods depends greatly on both the parameters specified for your model and the dataset you use to train it. So before making a final choice on the ML or DL model for your speaker recognition project, it would be best to test several models on different datasets to find the best-performing combination.

Conclusion

Speech recognition is the core element of complex speaker recognition solutions and is commonly implemented with the help of ML algorithms and deep neural networks. Depending on the complexity of the task at hand, you can combine different speaker recognition technologies, algorithms, and tools to improve the performance of your speech processing system.

When in doubt, consult a professional! Apriorit’s AI development team will gladly assist you with choosing the right set of parameters, tools, datasets, and models for your AI project. Reach out to us using the form below, or leave a message in the chat window!

Speech Recognition using Neural Networks

Speech Recognition using Neural Networks

Mr. Hardik Dudhrejia

Department of Computer Engineering

G H Patel College of Engineering & Technology Vadodara, India

Mr. Sanket Shah

Department of Computer Engineering

G H Patel College of Engineering & Technology Anand, India

AbstractSpeech is the most common way for humans to interact. Since it is the most effective method for communication, it can be also extended further to interact with the system. As a result, it has become extremely popular in no time. The speech recognition allows system to interact and process the data provided verbally by the user. Ever since the user can interact with the help of voice the user is not confined to the alphanumeric keys. Speech recognition can be defined as a process of recognizing the human voice to generate commands or word strings. It is also popularly known as ASR (Automatic speech recognition), computer speech recognition or speech to text (STT). Speech recognition activity can be performed after having a knowledge of diverse fields like linguistic and computer science. It is not an isolated activity. Various techniques available for speech recognition are HMM (Hidden Markov model)[1], DTW(Dynamic time warping)- based speech recognition[2], Neural Networks[3], Deep feedforward and recurrent neural networks[4] and End-to-end automatic speech recognition[5]. This paper mainly focusses on Different Neural networks used for Automatic speech recognition. This research paper primarily focusses on different types of neural networks used for speech recognition. In addition to this paper also consist of work done on speech recognition using this neural networks.

Keywords Speech recognition; Recurrent Neural network; Hidden Markov Model; Long Short term memory network

1. INTRODUCTION

   Throughout their life-span humans communicate mostly through voice since they learn all the relevant skills in their early age and continue to rely on speech communication. So, it is more efficient to communicate with speech rather than by using keyboard and mouse. Voice Recognition or Speech Recognition provides the methods using which computers can be upgraded to accept speech or human voice assist input instead of giving input by keyboard. It is extremely advantageous for the disabled people.

   Speech is affected greatly by the factors such as pronunciations, accents, roughness, pitch, volume, background noise, echoes and gender. Preliminary method of speech processing is the process of studying the speech signals and the methods of processing these signals.

   The conventional method of speech recognition insist in representing each word by its feature vector and pattern matching with the statistically available vectors using neural networks. On the contrary to the antediluvian method HMM, neural networks does not require prior knowledge of speech process and do not need statistics of speech data. [3]

   Types of speech recognition: Based on the type of words speech recognizing systems can recognize, the speech recognition system is divided into the following categories:

   • Isolated Word:

     Isolated word requires each utterance to have quiet on both sides of sample window. At a time only single words and single utterances are accepted and it is having Listen and Non-Listen state.

   • Continuous Word:

     Continuous speech recognisers provide the users a facility to speak in a continuous fashion and almost naturally and at the same time the computer determines the content of the speech. Recognisers rendering the facility of continuous speech capabilities are pretty much difficult to create because they require some special and peculiar methods in order to determine the boundaries of the utterances.

   • Connected Word:

     Connected words are very much alike the isolated words but they allow separate utterances to be executed with minimal pauses in between them.

   • Spontaneous speech:

   At an elementary level, spontaneous speech can be considered as a speech that is coming out naturally and not a rehearsed one. An Automatic Speech Recogniser must be able to handle a wide range of speech features like the words being run together.

   Classification of speech sounds:

   In this modern time, the process of classification of speech sounds is commonly done on the basis of 2 process based on how the classification process is looked upon:

   Based on the process of obstruction and non-obstruction sounds

   The process of classifying the sounds with respect to the process of obstruction and non-obstruction relies upon the conception of bodily air. While generating human sounds, the air coming out of the body has two functions; it is obstructed in the mouth or throat somewhere or it doesnt get obstructed, but the air comes out very easily. Correspondingly, the sounds that are produces as a result of obstructions and non-obstructions are not same excluding some of their qualities that are trivial.

   For e.g.- all the vowels (a,e,i,o,u) are non obstruction speech sounds and all the consonants (b,c,d,f,g,h,j,k,l,m,n,p,q,r,s,t,v,w,x,y,z) are obstruction speech sounds.

   Based on the process of voice and voiceless sounds

   Voiced sound is produced when the vocal chords vibrate when the sound is produced. Whereas in the voiceless sound no vocal cord vibration is produced. To test this, place your finger on your throat as you say the words. A vibration will be felt when the voiced sounds are uttered and no vibration will be felt while uttering a voiceless sound. Many a times it is difficult to feel the difference between them. So in order to distinguish between them another test can be performed by putting a paper in front of our mouth and the paper should move only by saying the voiceless sounds. All the vowels are voiced whereas some of the consonants are voiced as well as voiceless.

   Voiced consonants are :- b,d,g,v,z,th,sz,j,l,m,n,ng,r,w,y Voiceless consonants are :- p,t,k,f,s,th,sh,ch,h

2. SPEECH RECOGNITION PROCESS

   Speech Recognition is truly a ponderous and tiresome process. It consists of 5 steps:-

   1. Speech

   2. Speech Pre-Processing

   3. Feature Extraction

   4. Speech Classification

   5. Recognition

      numerous tools and softwares available which record the speech delivered by the humans. The phonic environment and the equipment device used have a significant impact on the speech generated. There is a possibility of having background or room reverberation blended with the speech but this is completely undesirable.

      Speech Pre-Processing

      The solution of the problem described above is the Speech Pre-Processing. It plays an influential role in cancelling out the trivial sources of variation. The speech pre-processing typically includes reverberation cancelling, echo cancellation, windowing, noise filtering and smoothing all of which conclusively improves the accuracy of speech recognition.

      Feature Extraction

      Each and every person has different speech and different intonation. This is due to the different characteristics ingrained in their utterance. There should be a probability of identifying speech from the theoretical waveform, at least theoretically. As a result of an enormous variation in speech there is an imminent need to reduce the variations by performing some feature extraction. The ensuing section depicts some of the feature extraction technologies which are extremely used nowadays.

      LPC (Linear Predictive Coding):- It s an extremely useful speech analysis technique for encoding quality speech at low bit rate and is one of the most powerful method. The key idea behind this method is that a specific speech sample at current time can be approximated as a linear combination of past speech samples. In this method the digital signal is compressed for competent storage and transmission. The principle behind the use of LPC is to reduce the sum of squared distance between the original speech and estimated speech over a finite duration. It can be further used to provide unique set of predictor coefficients. Gain (G) is also a crucial parameter.

      MFCC (Mel Frequency Cepstarl Coefficients):- This is the standard method feature extraction. It is preliminary based on the frequency domain which is based Mel scale based on human ear scale. They are more accurate than time domain features ever since they fall into the category of frequency domain features. The most conspicuous impediment is its sensitivity to noise as it is highly dependent on the spectral form. Techniques utilizing the periodicity of speech signals could be used to overcome this drawback although speech also encompasses aperiodic content.

      Speech

      Figure-1 Speech recognition process

      Speech Classification

      These systems are used to extract the hidden information from the input processing signals and comprises of convoluted mathematical functions. This section describes some commonly used speech classification techniques in brief.

      Speech is defined as the ability to express ones thoughts and feelings by articulate sounds. Initially the speech of a person is received in the form of a waveform. Also there are

      HMM (Hidden Markov Model):- This is the most strongly

      used method in order to recognize pattern in the speech. It is safer and possesses a secure mathematical foundation as

      compared to the template based and knowledge based approach. In this method, the system being modelled is assumed to be a Markov process having hidden states. The speech is distributed into smaller resounding entities each of which represent a state. In simpler Markov Model, the states are clearly visible to the user and thus the state transition probabilities are only the parameters. On the other hand, in hidden Markov Model, the state is not directly visible, but the output, which is dependent on the state, is evident. HMM are specifically known for their application in reinforcement learning and pattern recognition such as speech, handwriting and bioinformatics.

      DTW (Dynamic Time Warping):-In time series analysis, DTW is a kind of algorithm which measures the similarity or affinity between two temporal sequences that vary in speed or time. It correlates the speech words with reference words. This method changes the time dimension of the undiscovered words unless and until they are matched with the reference word. A well-known application of DTW is the automatic speech recognition, in order to cope up with different speaking speeds. Various other applications are online signature recognition and speaker recognition.

      VQ (Vector Quantization):- This method is preliminary based on block coding principle. This technique allows the modelling of probability density functions by the circulation of prototype vectors. It was formerly used for data compression. It performs the mapping of the vector from a vast vector space to a finite number of regions in that space. Every region is known as cluster and can be depicted by its centre called code word. Vector Quantization is used in lossy data compression, lossy data correction, and clustering and pattern recognition.

      Recognition

      After the above four phases speech recording, speech pre- processing, feature extraction and speech classification the final step that is remaining is the speech recognition. Once all the above mentioned steps are completed successfully then the recognition of speech can be done by three approaches.

      1. Acoustic phonetic approach[6]

      2. Pattern recognition approach[7]

      3. Artificial intelligence approach

      This paper is mainly concern with Artificial intelligence approach for speech recognition. This is a combination of acoustic phonetic and pattern recognition approach. In this approach system created by neural networks are used to classify and recognize the sound.. Neural networks are very powerful for recognition of speech. There are various networks for this process. RNN, LSTM, Deep Neural network and hybrid HMM-LSTM are used for speech recognition.

3. NEURAL NETWORKS

   Traditionally neural networks referred to as neurons or circuit. At present the term neural networks refers to as Artificial Neural Network, consisting of artificial neurons or

   nodes. It is a network of elementary elements known as artificial neurons, which receives an input, changes the state according to that input and generates an output. An interconnected group of natural or artificial neurons uses a mathematical model for information processing based on connectionist approach to communication. Neutral Networks can be treated as simple mathematical models defining a function f: X Y or a distribution over X or both X and Y [8], but many a times models are intimately connected with a particular learning rule. The term artificial neural network refers to inter-connections in among the neurons in the different layers of each system. Mathematically a neurons network function f(x) is defined as a composition of other functions gi(x), which can further be defined as a composition of other functions. A most commonly used type of composition is the nonlinear weighted sum, where f(x) = K (i ,wigi(x)), where K is a predefined function. Referring the collection of functions gi as a vector g simply would be more advantageous. The first view being the functional view; the given x is then changed to a 3-D vector h, which is then finally transformed into f. This view is frequently encountered in the optimization context. Probabilistic view is the second view and the arbitrary variable G = g (H), depends on H = h(X), which in turn depends upon the X(random variable). In the perspective of graphical models this view most commonly comes across. Networks described such as the above one are often called feed forward as its graph is a DAG (Directed Acyclic Graph). Networks which have cycles in it is called recurrent neural networks.

   Neural Network Models:

   Language modelling and acoustic modelling are both vital aspects of modern statistically-based speech recognition systems. The ensuring section illustrates various methods for Speech Recognition.

   Deep Neural Network (Hidden Markov Models HMM is a generative model in which observable acoustic features are assumed to be generated from a hidden Markov process that transitions between states S = {s1, s2 ., sk}.

   HMM was the most widely used technique for Large Vocabulary Speech Recognition (LSVR) for at least two decades. The decisive parameters in the HMM are the initial state probability distribution f = {p(q0=si)}, where qt is the state at time t; the transition probabilities aij= p(qt=sj |qt- 1=si) ; and a model to estimate the observation probabilities p(xt |si). The conventional HMM used in the process of Automatic Speech Recognition had their observation probability modelled by using Gaussian Mixture Model (GMM). Even the GMM had a vast number of advantages, the issue was that, they were statistically inept for modelling data that lie on or near the non-linear diverse in the space. For instance, modelling of the points residing very close to surface of the sphere hardly requires any parameter using a suitable model class, but it requires large number of diagonal Gaussians or a fairly huge number of full- covariance Gaussians. Because of this other types of models may work better than GMM the exploitation of information embedded in a large window of frames is done well. On the

   other hand ANN (Artificial Neural Network) can handle the data resiing on or near a non-linear model more effectively and learn much better models of data. Since the past few years, outstanding advances have been made both in machine learning algorithms and computer hardware which ultimately has led to more efficient methods of learning having many layers and a large layer of output. The output layer must hold a great number of HMM states that arise on each phone being modelled by a different number of triphone. By employing various new learning methods, vast number of research group have shown that Deep Neural Network has better performance than GMMs at acoustic modelling for recognition of speech that includes massive vocabularies and behemoth dataset. An Artificial Neural Network has more than one layer of hidden units between its inputs and outputs whereas Deep Neural Network is a feed-forward network. Each hidden unit in DNN, j, uses the logistic function to map all of its input from the layer below, xj , to the scalar state, yj that it sends to the above lying layer.

   Yj = log (xj) = 1/ (1+e-xj)

   Xj = bj+ yiwij

   i

   Where bj is the bias of unit j, i is an index over units in the layer below, and wijis the weight on a connection to unit j from unit i in the layer below. For multiclass classification, output unit j converts its total input, xj , into a class probability, pj, by using the softmax non-linearity

   pj= exp(xj) / exp(xk)

   k

   where k is an index over all classes.

   Deep Neural Network DNNs can be trained by back- propagating derivatives of a cost function that measures the divergence between the target outputs and the actual outputs produced. When working with softmax function, the natural cost C can be calculated as

   C = – djlogpj

   j

   where p is the output of the softmax d represents the target probabilities.

   DNNs having many hidden layers are difficult to optimize. The optimal way is not to choose the gradient descent from a arbitrary starting point near the origin in order to find a good set of weights and unless the initial scales of the weight are carefully chosen, in different layer there will be varying magnitudes of the back propagated gradients. In addition to these issues, generalization of test data may be done poorly by DNN. Layers of DNN are quite flexible and each with a large number of parameters. As a result of this it makes DNN capable of modelling very intricate and non- linear relationships between outputs and inputs. There is a possibility of having severe over fitting and this issue can be eliminated by early stopping or weight penalties but it is only possible by reducing considerable amount of power. Large amount of dataset can reduce the over fitting and

   preserving the modelling power simultaneously but it increases the computational cost. Therefore there is a prominent need of using the information in the process of training set to build various layers of non-linear feature detectors.

   Recurrent Neural Network: Recurrent Neural Network is a kind of Artificial Neural Network, which is represented in the form of directed cycle where each and every node is connected to the other nodes. Two units become dynamic as soon as the communication takes place in between them. Since RNN uses internal memory just like the feed-forward networks to process the sequence of arbitrary input, it makes them ideal choice for speech recognition. The key feature of RNN is that the activations flow round in a loop as the network contains at least one feed-back connection thus allowing the networks to do temporal processing and learn sequences.

   Figure 1 Recurrent Neural Network[9]

   The elementary structure is a feed-forward DNN having an input and output layer and certain number of hidden layers with full recurrent connections. Even of basic architectures, learning in RNN can be achieved. In Fig. 2 representation of simplest form of fully recurrent neural network that is an MLP (Multi-Layer Preceptron) having previous set of concealed unit activations (h (t)), feeding back into the network along with the input (h (t+1)). The time scale t refers to the operation of real neurons and as far as artificial systems are concerned any relevant time step size for the given problem can be used. In order to hold the unit until they are processed at the next step, a delay unit is introduced purposely.

   LSTM (Long Short Term Memory): LSTM is a RNN architecture that in addition to regular network unit, contains LSTM blocks. LSTM blocks are generally referred to as smart network unit that possess the capability of remembering a value having arbitrary length of time. It contains gates whose function is to let us know when the input is significant to remember, when to forget and when it should output the value. In LSTM architecture, a set of recurrently connected subnets known as memory blocks resides in the recurrent hidden layer. In order to control the

   flow of information, each memory block contains one or more self-connected memory cells and three multiplicative gates. The flow of information in each cell of LSTM is secured by the learned input and output gates. The forget gate is added for the purpose of resetting the cells. A conventional LSTM can be defined as follows:

   Given an input sequence x = (x1, x2 , . . . , xT ), a conventional RNN computes the hidden vector sequence h = (p, p, . . . , hT ) and output vector sequence y = (y1, y2, . .

   . , yT ) from t = 1 to T as follows:

   ht = H (Wxhxt +Whhht1 + bh) t = Whyht + by

   Where, the W denotes weight matrices, the b denotes bias vectors and H(·) is the recurrent hidden layer function. The following figure illustrates the architecture of LSTM:-

   Figure 2 Architecture of LSTM network having a single memory block[10]

   Limitations of LSTM:

   Because LSTM has the capacity to store only one of its inputs, error will not be reduced which in turn wont moderate the error by solving the sub goal. Full gradient can be used as a solution to this problem.

   1. However full gradient has also some limitations: 1) It is very complex 2) Error flow is visible only when truncated LSTM is used.

   2. The weight factor increases to 32 as a single hidden unit is replaced by 3 units. Therefore, a single memory has the requirement of two additional cell blocks.

   3. The problems associated with the feed forward nets also persists in the LSTM because, it behaves like fed forward neural network trained by back propagation neural network to see the complete input string.

   4. Practical problem in all the gradient based approaches is Counting the time steps problem.

      Advantages of LSTM:

      1. The constant error back propagation in LSTM allows it to bridge extremely long time lags in case of similar problems discussed above. In case of

         long time lags, LSTM can handle noise, continuous values and distributed representations. However in case of hidden Markov Model LSTM doesnt need to have priori choice of a finite number of states, in that it can deal with infinite state numbers.

      2. With respect to the problems discussed in this paper, LSTM can generalize well irrespective of the irrelevant and widely spread inputs in the input sequence. It has the capability of quickly learning about how to differentiate in between two or more widely spread occurrences of a particular element in the input sequence without getting dependent on short time lag training exemplars.

      3. It doesnt require parameter tuning at all. It works pretty well with wide range of parameters such as input and output bias gate, learning rate.

      4. The LSTMs algorithm complexity per weight and time step is O(1). This is considered to be extremely advantageous and outruns the other approaches such as RTRL.

4. LITERATURE SURVEY

   In early 1920s, speech recognition came into existence. The first machine to recognize speech is named Radio Rex. (Manufactured in 1920). After that, research is begun in Bell Labs in 1936[12.In 1939, Bell labs demonstrated a speech synthesis machine at the world Fair in New York. In 1952 three Bell Labs researchers, Stephen.Balashek , R. Biddulph, and K. H. Davis built a system called Audrey an automatic digit recognizer for single-speaker digit recognition. Their system worked by locating the formates in the power spectrum of each utterance. [13] The 1950s era technology was limited to single-sAupeaker systems with vocabularies of around ten word. Michael Price, James Glass, Anantha P. Chandrakasan (2015) [14] described an IC that provides a local speech recognition capability for a variety of electronic devices. With 5,000 word recognition tasks in real-time with 13.0% word error rate, 6.0 mW core power consumption, chip provides search efficiency of approximately 16 nJ per hypothesis. The vowel recognizer of Forgie and Forgie constructed at MIT Lincoln laboratories in 1959 in which 10 vowels embedded in a /b/- vowel/t/ format were recognized in a speaker independent manner.[15]

   In late 1960s Raj Reddy was first to take on continuous speech recognition at Stanford university. Early systems are based on pause between each word. This is first continuous speech recognition approach. In the meanwhile, Soviet Union has used DTW algorithm to build a 200-word vocabulary speech recognition machine.

   In 1970s, Velichko and Zagoruyko have studied discrete utterance recognition or isolated word in Russia.[17] Same in United states by Itakura and in Japan by Cakoe and Chiba[18].In 1971, striving speech understanding project was funded by Defence Advanced Research Projects Agencies (DARPA). IEEE acoustic, Speech, and Signal Processing group held a conference in Newton, Massachusetts in 1972.

   In mid 1980s, IBM created a voice-activated typewriter called Tangora, which could handle a 20,000-word vocabulary under the lead of Fred Jelinek. [16] In this era, neural networks are emerged as an attractive model for Automatic speech recognition. Speech research in the 1980s was shifted to statistical modelling rather than template based approach. This is mainly known as Hidden Markov model approach. Applying neural networks for speech recognition was reintroduced in late 1980s. Neural networks first introduced in 1950 but for some practical problems they were not that much efficient.

   In the 1990s, the Bayes classification is transformed into the optimization problems, which also reduces the empirical errors. A key issue in the design and implementation of speech recognition system is how to choose proper method in the speech material used to train the recognition algorithm. Training can be supervised learning in which class is labelled in the training data and algorithm will predict the label in the unlabelled data. Stephen V. Kosonocky [11] had researched about how neural network can be used for speech recognition in 1995.

   In 2005, Giuseppe Riccardi [19] developed Variational Bayesian (VB) to solve the problem of adaptive learning in speech recognition and proposed learning algorithm for ASR.

   In 2011, Dr.R.L.K.Venkateswarlu, Dr. R. VasanthaKumari, G.VaniJayaSri[20]utilizes Recurrent Neural Network, one of the Neural Network techniques to observe the difference of alphabet from E- set to AH – set. In their research 6 speakers (a mixture of male and female) are trained in quiet environment. The English language offers a ndreumber of challenges for speech recognition. They used multilayer back propagation algorithm for training the neural network. Six speakers were trained using the multilayer perceptron with 108 input nodes, 2 hidden layers and 4 output nodes each for one word, with the noncurvear activation function sigmoid. The learning rate was taken as 0.1, momentum rate was taken as 0.5.Weights were initialized to random values between +0.1 and -0.1 and accepted error was chosen as

   0.009. They have compared the performance of neural network with Multi-Layer Perceptron and concluded that RNN is better than Multi-Layer Perceptron. For A-set the maximum performances of speakers 1-6 were 93%, 99%, 96%, 93%, 92% & 94%. For E-set it was 99%, 100%, 98%, 97%, 97% & 95%, and For EH-set 100%, 95%, 98%, 95%, 98% & 98% and lastly for AH-set 95%, 98%, 96%, 96%, 95% & 95% respectively. Results shows that RNN is very powerful in classifying the speech signals.

   Song, W., &Cai, J. (2015) [21] has developed end to end speech recognition using hybrid CNN and RNN. They have used hybrid convolutional neural networks for phoneme recognition and HMM for word decoding. Their best model achieved an accuracy of 26.3% frame error on the standard core test dataset for TIMIT. Their main motto is to replace GMM-HMM based automatic speech recognition with the deep neural networks. The CNN they used consists of 4

   convolutional layers. The first two layers have max pooling and the next two densely connected layers with a softmax layer as output. The activation function used was ReLu. They implemented a rectangular convolutional kernel instead of square kernel.

5. CONCLUSION

   Speech is primary and essential way for communication between humans. This survey is about neural networks are modern way for recognizing the speech. In contrast to traditional approach it does not requires any statistics.A speech recognition system should include the four stages: Analysis, Feature Extraction, Modeling and matching techniques as described in the paper. In this paper, the fundamentals of speech recognition are discussed and its recent progress is investigated. Various neural networks model such as deep neural networks, and RNN and LSTM are discussed in the paper. Automatic speech recognition using neural networks is emerging field now a day. Text to speech and speech to text are two application that are useful for disabled people. Paper mainly focuses on speech recognition of one language, which is English.

6. REFERENCES

[1] Yu D., Deng L. (2015) Deep Neural Network-Hidden Markov Model Hybrid Systems. In: Automatic Speech Recognition. Signals and Communication Technology. pp 99-116, Springer, London,[Available Online]: Automatic Speech Recognition Using HMM and deep neural network

[2] Zhang, XL, Luo, ZG. & Li, M. J, Journal Of Computer Science and Technology, Springer ,November 2014, Volume 29, Issue 6, pp 10721082.https://doi.org/10.1007/s11390-

014-1491-0

[3] Zou J., Han Y., So SS. (2008) Overview of Artificial Neural Networks. In: Livingstone D.J. (Eds) Artificial Neural Networks. Methods in Molecular Biology, vol 458. Humana Press, [Available Online]: https://link.springer.com/protocol/10.1007/978 -1-60327- 101-1_2

[4] Recurrent deep neural networks for robust speech recognition.

/ Weng, Chao; Yu, Dong; Watanabe, Shinji; Juang, Biing Hwang Fred. IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP 2014. Institute of Electrical and Electronics Engineers Inc., 2014. p. 5532-5536 6854661.

[5]
[6]

Sciences), vol 46. Springer, Berlin, Heidelberg

[7] Rabiner L.R. (1992) Speech Recognition Based on Pattern Recognition Approaches. In: Ince A.N. (eds) Digital Speech Processing. The Kluwer International Series in Engineering and Computer Science (VLSI, Computer Architecture and Digital Signal Processing), vol 155. Springer, Boston, MA

[8] Wikipedia contributors. (2018, September 22). Neural network. In Wikipedia, The Free Encyclopaedia. Retrieved 15:37, Novmber 4 , 2017, from https://en.wikipedia.org/w/index.php?title=Neural_netw ork&oldid=860697996

[9] International Journal on Recent and Innovation Trends in Computing and Communication Volume: 4

[10] G Gnaneswari, S R VijayaRaghava, A K Thushar, Dr.S.Balaji, Recent Trends in Application of Neural Networks to Speech Recognition, International Journal on Recent and Innovation Trends in Computing and Communication,

Volume: 4 Issue 1,pp 18 – 25

[11] Kosonocky S.V. (1995) Speech Recognition Using Neural Networks. In: Ramachandran R.P., Mammone R.J. (eds) Modern Methods of Speech Processing. The Springer International Series in Engineering and Computer Science (VLSI, Computer Architecture and Digital Signal Processing), vol 327. Springer, Boston, MA

[12] Wikipedia contributors. (2018, October 2). Bell Labs. In Wikipedia, The Free Encyclopedia. Retrieved 13:35, October 5, 2018, from https://en.wikipedia.org/w/index.php?title=Bell_Labs&o ldid=862196483

[13] Juang, B. H.; Rabiner, Lawrence R. «Automatic speech recognitiona brief history of the technology development» (PDF): 6. Archived (PDF) from the original on 17 August 2014. Retrieved 17 January 2015

[14] Michael Price, James Glass, Anantha P. Chandrakasan A 6 mW, 5,000-Word Real-Time Speech Recognizer Using WFST Models, IEEE Journal Of Solid-State Circuits, Vol. 50, No. 1, PP 102-112, January 2015

[15] W. Forgie, James & D. Forgie, Carma. (1959). Results Obtained from a Vowel Recognition Computer Program. The Journal of the Acoustical Society of America. 31. 844-844. 10.1121/1.1936151.

[16] Pioneer speech recognition, [available online]: http://www03.ibm.com/ibm/history/ibm100/us/en/ico ns/speechreco/

[17] V. M. Velichko and N. G. Zagoruyko, Automatic recognition of 200 words, Int. J. Man-Machine Studies, 2:223, June 1970

[18] H. Sakoe and S. Chiba, Dynamic programming algorithm optimization for spoken word recognition, IEEE Trans. Acoustics, speech, signal proc., ASSP 26(1), pp. 43-49, Febreary 1978.

[19] Giuseppe Riccardi, DilekHakkani-Tür, «Active learning: theory and applications to automatic speech recognition», IEEE Transaction on Speech and Audio Processing, vol. 13, no. 4, pp. 504-511, 2005.

[20] Dr.R.L.K.Venkateswarlu, Dr. R. Vasantha Kumari, G.VaniJayaSri, International Journal of Scientific & Engineering Research Volume 2, Issue 6, June-2011

[21] Song, W., &Cai, J. (2015) End-to-End Deep Neural Network for Automatic Speech Recognition

Sanket A. Shah was born in Anand City, Gujarat, India in 1997. He is currently pursuing his computer engineering at

G.H. PATEL INSTITUTE OF ENGINEERING AND TECHNOLOGY, Bakrol, Gujarat, India. He has attended the national conference, RACST held at his institute in 2016.

Hardik J. Dudhrejia was born in Rajkot City, Gujarat, India in 1997. He is currently pursuing his computer engineering at G.H. PATEL INSTITUTE OF ENGINEERING AND TECHNOLOGY, Bakrol, Gujarat,

India. He has attended various workshops as well as conferences including a national conference, IMPRESSARIO at his institute in 2016.

.