Projects:2015s1-05 Multi-Profile Parallel Transcriber

Team

Students

• Dakshitha Narendra Kirana Kankanamage
• Siyuan Ma

Supervisors

• Dr. Said Al-Sarawi
• Dr. Ahmad Hashemi-Sakhtsari

Introduction

The field of Automatic Speech Recognition (ASR) has been a major field of interest for researchers over the past few decades. Although, it was strongly believed to play a major role in Human-Machine as well as Human-Human interactions [1] [2] [3], the real world users have been unable to utilize this due to the slow pace of advancements in the field. Today, with the rise of portable electronic devices and more powerful computers, Speech Recognition (SR) technology has experienced a great push forward, thereby providing the everyday user a better Human-Machine experience and researchers the power to overcome obstacles that could not have been done so before.

Motivation and Significance

The Multi-Profile Parallel Transcriber prototype is one such example of an ASR, which was custom made as a Speech-To-Text (STT) transcriber in the hopes of demonstrating speaker-independent transcription, created by the use of a speaker-dependent speech recognizer, Dragon NaturallySpeaking (DNS). While its main aim is to transcribe the speech of unknown speakers, it also retains vital information regarding the transcription such as time stamps and the conversation of each speaker in a separate audio file, while providing the flexibility to edit errors in the transcripts. Due to its ability to identify and retain vital information from a conversation, it was found to play an important part in various settings that conventionally requires a human transcriber, including but not limited to situations such as meetings, brain storming sessions and interviews.

Ability to do a long-time transcribing with a satisfying accuracy is one significant of advantages of the Multi-Profile Parallel Transcriber. Another advantage of the Multi-Profile Parallel Transcriber is that unlike Siri or some other applications, it is able to achieve off-line transcription which will assure the security of the information.

Aims

This is a DSTO sponsored project and it aims to evaluate, upgrade and extend the functionality of the speech-to-text software system known as Multi-Profile Parallel Transcriber. The stages of this project can be summarized as follow: 

• In the first stage, the Multi-Profile Parallel Transcriber will be set-up following the instructions in the provided documents.

• This software then will be ran through an evaluation phase where the Multi-Profile Parallel Transcriber will be tested against various benchmark databases.

• Then the core software called Dragon Naturally Speaking will be upgraded to the latest version.

• Following the upgrade, the same evaluation phase as the first stage will be implemented again.

• At the end of the evaluation phase, the system will then have its functionality extended. This includes increasing the efficiency of the software and implementing a different approach as to how the software works.

Requirements

The core requirements to solve the proposed questions of the project are decided as follow:

• The first task is to set up a working copy and evaluate the performance of the current version of the Multi-Profile Parallel Transcriber prototype.

• Then we need to upgrade the current version of the Multi-Profile Parallel Transcriber prototype, which uses DNS version.10, to DNS version.13, which is the latest version of DNS, with the help of the source codes provided and the API information from the DNS SDKs provided.

• The upgraded software shall be tested using formal methods of evaluation with the help of tools named SCLite. If there is a need to improve the results of the system, this shall be done.

• Finally, if time permits, a solution to overlapping speech of speakers and being able to find an alternative to having VMs to be implemented.

System Overview

The typical architecture of an ASR system consists of 4 main elements: Signal processing module, Acoustic Model, Language Model and a Hypothesis Search module [1] [3], as shown in Figure 1.

• The signal processing component takes an input audio signal and enhances the speech information by reducing noise and converting the signal from time-domain to frequency-domain for easier processing.

• The Acoustic Model, takes the frequency domain signal and generates an Acoustic Model score of length of the input vector. An Acoustic Model consists of information regarding acoustics, phonetics, environmental variability, gender and other aspects that would affect a person’s speech.

• The Language Model estimates the probability of a word occurring in a sequence of spoken words, this is achieved with the help of a database of text (corpora).

• The last stage of the process, the Hypothesis Search, combines both Acoustic and Language model scores and gives the sequence of words with the highest score as the recognised result known as a confidence score [3] [1, 4].

As each speaker’s speech style is unique, the default Acoustic Model or Language Model is modified against the speaker’s speech style prior transcription to improve accuracy and performance. Modifying these models according to a user is known as training. Through training, a profile for the user will be created and this can be attained by having the speaker read a known passage of text or having to manually transcribe a voice recording of the speaker with a valid transcript for a few minutes.

For this project, with the aim of transcribing an incoming signal, the Multi-Profile Parallel Transcriber consists of two main components, namely a Streaming Host program and several Streaming Guest programs. The Streaming Host would reside on the operating system of the host computer, having control over multiple Streaming Guests as well as having access to a shared directory with a sub-directory for all Guest instances, which will store all vital information of the transcriptions. All Streaming Guests need to be integrated with an instance of DNS speech recognition engine but since DNS can only be instantiated once on a computer, each guest machine has to be run through a Virtual Machine (VM) in order to simultaneously run multiple guests[As shown in Figure 2]. DNS is speaker-dependent speech transcriber, therefore the need to have a speech profile assigned for each guest instance plays a vital part for the project.

The prototype gets the structure of an ASR by having the Streaming Host receive a digital audio input signal, the signal is then distributed to all Streaming Guest profiles. These Guest instances transcribe the signal against a speech profile, allocated by the Streaming Host, and return the transcribed information back to the Streaming Host with confidence scores for each utterance. The Host will then assemble the best confidence score as the best match for the speaker. DNS is speaker-dependent speech transcriber, therefore the need to have a speech profile assigned for each guest instance plays a vital part for the project.

Apart from being the mediator, the Host machine also has the functionality of changing the settings of each guest profile, these include: changing the speaker profiles, being able to correct errors in the transcription and change parameters of the guest VMs such as the interval between utterances.

Knowledge Gaps and Technical Challenges

Related to the tasks of the project, some knowledge gaps are described as below:

• As the main workload of this project is evaluating the performance of Multi-Profile Parallel Transcriber, the formal evaluation tools and standards need to be used and learned for some stage of the project. Before and after the upgrades have been implemented, the system should be tested and validated with Speech Recognition scoring tools such as SCLite. This will require an overall knowledge of how these software needs to be used and important standards that needs to be met [7-9].

• Another aspect of this project is to make improvements to Multi-Profile Parallel Transcriber. The 1st improvement is to upgrade DNS version.10 to DNS version.13. As the source codes for DNS is programmed using Delphi, it is a more straightforward approach to continue using Delphi. Since knowledge about Delphi is not covered by the general learning content in the university, learning how to program basic Delphi would be the first knowledge gap needs to be fulfill.

• The use of DNS Software Development Kit (SDK) would also be required to make the upgrade from DNS 10 to 13, therefore having prior knowledge of how to use the SDK will also be useful.

• If time permits, further upgrades of the system shall be done, this would include having the system work under overlapping speech and also an alternative to the VMs implementations.

The possible technical challenges are closely connected to knowledge gaps listed above, so overcoming the knowledge gaps would be considered the main technical challenges the group will have to face.

Project Status

Considering the proposed requirements of the project was made in the middle of the semester, the progress is behind the schedule. Some adjustments have been made to assure that it is possible to achieve the basic requirements in the end of the project.

According to the new plan, the next step will be still working on the recent tasks, Daki’s task is to find the problems with shared folder and fix them with the support from DSTO and the university. With a new arrangement to build the system, a working copy has been made and work normally.

Siyuan' task is to learn Delphi and recognize the functions related to DNS in the source code and using the API information from the different versions of DNS SDK to figure out whether the names of the functions have been changed or not. This is going to be a basic assurance that the software can be upgraded successfully. For now, all the functions used have been summarize in a separated document with all the related information and the result is that all the functions have been matched.

Resources

• Delphi XE3
• Dragon NaturallySpeaking (version: 9, 10, 10.1, 11, 12, 13)
• Dragon NaturallySpeaking SDK (version: 10, 11, 12)
• VmWare (Virtual Machines)
• Sclite by NIST (National Institute of Standards and Technology)

References

[1] L. D. Dong Yu, Automatic Speech Recognition: A Deep Learning Approach, 1 ed. London: Springer London, 2015.

[2] P. M. Kitzing, A ; Ahlander, Vl, "Automatic speech recognition (ASR) and its use as a tool for assessment or therapy of voice, speech, and language disorders," Logopedics Phoniatrics Vocology, vol. 34, pp. 91-96, 2009.

[3] X. H. a. L. Deng, "An Overview of Modern Speech Recognition," in Handbook of Natural Language Processing, Second Edition, N. I. a. F. J. Damerau, Ed., ed Boca Raton, FL: Chapman & Hall/CRC, 2010.