Projects:2018s1-103 Improving Usability and User Interaction with KALDI Open-Source Speech Recogniser

Project Team

Students

• Shi Yik Chin
• Yasasa Saman Tennakoon

Supervisors

• Dr. Said Al-Sarawi
• Dr. Ahmad Hashemi-Sakhtsari (DST Group)

Abstract

This project aims to refine and improve the capabilities of KALDI (an Open Source Speech Recogniser). This will require:

• Improving the current GUI's flexibility
• Introducing new elements or replacing older elements in the GUI for ease of use
• Including a methodology that users (of any skill level) can use to improve or introduce Language or Acoustic models into the software
• Refining current Language and Acoustic models in the software to reduce the Word Error Rate (WER)
• Introducing a neural network in the software to reduce the Word Error Rate (WER)
• Introducing a feedback loop into the software to reduce the Word Error Rate (WER)
• Introducing Binarized Neural Networks into the training methods to reduce training times and increase efficiency

This project will involve the use of Deep Learning algorithms (Automatic Speech Recognition related), software development (C++) and performance evaluation through the Word Error Rate formula. Very little hardware will be involved through its entirety.

Introduction

KALDI is an open source speech transcription toolkit intended for use by speech recognition researchers. The software allows the utilisation of integration of newly developed speech transcription algorithms. The software usability is limited due to the requirements of using complex scripting language and operating system specific commands. In this project, a Graphical User Interface (GUI) has been developed, that allows non-technical individuals to use the software and make it easier to interact with. The GUI allows for different language and acoustic models selections and transcription either from a file or live input – live decoding. Also, two newly trained models have been added, one which uses Gaussian Mixture Models, while the second uses a Neural Network model. The first one was selected to allow for benchmark performance evaluation, while the second to demonstrate an improved transcription accuracy.

This project is a follow-on from previous year's project of the same name.

Background

The KALDI Toolkit

---

Initially developed in 2009 by Dan Povey and several others [20], KALDI has been aimed at being a low development cost, high quality speech recognition toolkit. Having undergone several improvements over the years, it now consists of recipes on creating custom Acoustic and Language Models that are based on several large corpuses such as the Wall Street Journal, Fisher, TIMIT etc. It's latest addition has been Neural Network Acoustic/Language models that are still in development [8].

KALDI's directory structure, and overall code layout is quite complex. It also requires the user to have a sound knowledge on Ubuntu terminal commands as well as knowledge on C++ and Python. It is, in fact, this complexity that the first-time users face when approaching the toolkit that has led to the inception of the previous iterations of the ASR program.

Qt Creator

In order to improve the GUI to suit the current iteration of the program, it was decided to use Qt Creator. This was also the GUI toolkit of choice for the previous students due to several reasons; it's open source, based on C++, backed by a large enterprise that updates the toolkit regularly, available on Linux and is simple to use.

Qt Creator includes a code editor and integrates Qt Designer which allows for designing and building GUI's from Qt widgets. It's also possible to compose and customize your own widgets as required. The developers of Qt Creator, Qt Project, have also ensured that a large library of documentation, tutorials, example code and FAQ’s is kept up to date with newer developments, bugs and bug fixes [28].

References

[1]-R.E. Gruhn, W. Minker, S. Nakamura, Statistical Pronunciation Modeling for Non-Native Speech Processing, ch. 2, p. 5, Harman/Becker Automotive Systems GmbH, Ulm, Germany: 2011 [E-book]. Available: Springer, https://goo.gl/peFMrZ [Accessed 22 May 2018]. [3]-J. Chiu. GHC 4012. Class Lecture, Topic: “Design and Implementation of Speech Recognition Systems – Feature Implementation”, School of Computer Science, Carnegie Mellon University, Pittsburgh, PA, 27 Jan. 2014. [4]-M. Breum, “Playing with Sounds in Words- Part 3 {Phoneme Segmentation}.” thisreadingmama.com, para. 3, 26 Sept. 2016. [Online]. Available: thisreadingmama.com/playing-with-phonemes-part-3/ [Accessed 22 May 2018]. [5]-“About Lexicons and Phonetic Alphabets (Microsoft.Speech)” Microsoft Research Blog, 2018. [Online]. Available: https://goo.gl/SDqZtU [Accessed 22 May 2018]. [6]-R. J. Mooney. CS 388. Class Lecture, Topic: “Natural Language Processing: N-Gram Language Models”, School of Computer Science, University of Texas at Austin, Austin, TX, 2018. [7]-A. Hannun, C. Case, J. Casper, B. Catanzaro, G. Diamos, E. Elsen, R. Prenger, S. Sateesh, S. Sengupta, A. Coates, A. Y. Ng, “Deep Speech: Scaling up end-to-end speech recognition”, p. 3, 19 Dec. 2014 [Abstract]. Available: https://arxiv.org/abs/1412.5567?context=cs [Accessed 22 May 2018]. [8]-KALDI, “Deep Neural Networks in Kaldi - Introduction”, KALDI, n.d. [Online] Available at: http://kaldi-asr.org/doc/dnn.html [Accessed 22 May 2018]. [20]-KALDI, “History of the KALDI Project”, KALDI, n.d. [Online] Available at: http://kaldi-asr.org/doc/history.html [Accessed 22 May 2018]. [28]- Qt, “Category:Tools::QtCreator”, Qt Wiki, 15 Sep. 2016 [Online]. Available: http://wiki.qt.io/Category:Tools::QtCreator [Accessed 22 May 2018]

Results

Improvement On GUI

Graphical User Interface

1. Editable transcript
   • An edit mode is added to the GUI. If the edit mode checkbox is checked, user will be able to edit the transcription directly from the display window. This allow the user to correct the transcription if a mistake is found.
2. Interactive display window
3. Ability to choose Acoustic and Language Models
   • Instead of having a fixed AM and LM, user is able to choose different AM and LM for their decoding session.
4. Audio recording & playback
5. Timestamp usage on utterances
6. Recording status indicator
7. Microphone mute/unmute control
8. Detailed documentation of decoding session

Improvement of Functionality of the ASR

New Acoustic and Language Models are added to the existing program:

1. Two new Acoustic Models
   • The first Acoustic Model is GMM a model. This is trained by using the corpus provided by LibriSpeech. The LibriSpeech corpus is a large (1000 hour) corpus of English read speech derived from audiobooks, sampled at 16kHz. The accents are various and not marked, but the majority are US English. Several models are trained from this corpus, which includes both monophone model and triphone model (tri2b, tri3b, ..., to tri6b).
   • The other Acoustic Model is a NNET2 model. This model is pre-trained using the more than 2000 hours of Fisher corpus and available for download from kaldi website.
2. A new Language Model
   • This Language Model is a 3-Gram LM and trained by a plain text of 34 MB

Detailed Documentation of decoding session

1. For every decoding session, all details from the session will be recorded and saved along with the transcription. These includes the Acoustic and Language Model used for the session, data and time of the session, and also whether a GMM or NNET2 model is used for the session.