tflite-micro/tensorflow/lite/micro/examples/micro_speech/train

Micro Speech Training

This example shows how to train a less than 20 kB model that can recognize 2 keywords, "yes" and "no", from speech data.

If the input does not belong to either categories, it is classified as "unknown" and if the input is silent, it is classified as "silence".

You can retrain it to recognize any combination of words (2 or more) from this list:

yes
no
up
down
left
right
on
off
stop
go

The scripts used in training the model have been sourced from the Simple Audio Recognition tutorial.

Table of contents

• Overview
• Training
• Trained Models
• Model Architecture
• Dataset
• Preprocessing Speech Input
• Other Training Methods

Overview

1. Dataset: Speech Commands, Version 2. (Download Link, Paper)
2. Dataset Type: Speech
3. Deep Learning Framework: TensorFlow 1.5
4. Language: Python 3.7
5. Model Size: <20 kB
6. Model Category: Multiclass Classification

Training

Train the model in the cloud using Google Colaboratory or locally using a Jupyter Notebook.

Google Colaboratory

Jupyter Notebook

Estimated Training Time: ~2 Hours.

For more options, refer to the Other Training Methods section.

Trained Models

Download Link	speech_commands.zip

The models directory in the above zip file can be generated by following the instructions in the Training section above. It includes the following 3 model files:

Name	Format	Target Framework	Target Device
model.pb	Frozen	TensorFlow	Large-Scale/Cloud/Servers

: : GraphDef : : :

| model.tflite | Fully | TensorFlow Lite | Mobile Devices |

(<20 kB) : Quantized* : : :

       : TFLite Model :                  :                           :

| model.cc | C Source | TensorFlow Lite | Microcontrollers |

       : File         : for              :                           :

       :              : Microcontrollers :                           :

*Fully quantized implies that the model is strictly int8 quantized including the input(s) and output(s).

Model Architecture

This is a simple model comprised of a Convolutional 2D layer, a Fully Connected Layer or a MatMul Layer (output: logits) and a Softmax layer (output: probabilities) as shown below. Refer to the tiny_conv model architecture.

This image was derived from visualizing the 'models/micro_speech_quantized.tflite' file in Netron

This doesn't produce a highly accurate model, but it's designed to be used as the first stage of a pipeline, running on a low-energy piece of hardware that can always be on, and then wake higher-power chips when a possible utterance has been found, so that more accurate analysis can be done. Additionally, the model takes in preprocessed speech input as a result of which we can leverage a simpler model for inference results.

Dataset

The Speech Commands Dataset. (Download Link, Paper) consists of over 105,000 WAVE audio files of people saying thirty different words. This data was collected by Google and released under a CC BY license. You can help improve it by contributing five minutes of your own voice. The archive is over 2GB, so this part may take a while, but you should see progress logs, and once it's been downloaded you won't need to do this again.

Preprocessing Speech Input

In this section we discuss spectrograms, the preprocessed speech input to the model.

The model doesn't take in raw audio sample data, instead it works with spectrograms which are two dimensional arrays that are made up of slices of frequency information, each taken from a different time window.

The recipe for creating the spectrogram data is that each frequency slice is created by running an FFT across a 30ms window of the audio sample data. The input samples are treated as being between -1 and +1 as real values (encoded as -32,768 and 32,767 in 16-bit signed integer samples). The audio sampling window stride is 20ms, thus every window overlaps by 10ms.

This results in an FFT with 257 entries. Every sequence of approximately six entries is averaged together, giving a total of 40 frequency buckets in the slice. The results are further processed by down-scaling, noise reduction, automatic gain control, and a final down-scaling.

Each adjacent frequency entry is stored in ascending memory order (frequency bucket 0 at data[0], bucket 1 at data[1], etc). The window for the frequency analysis is then moved forward by 20ms, and the process repeated, storing the results of the new frequency slice in the next memory row. The training is configured for raw audio samples of 1000ms in length. With a window size of 30ms and stride of 20ms, some 49 frequency slices can be created from 1000ms of audio data. Thus, the preprocessing produces a single channel image that is 40 pixels wide, and 49 rows high.

Other Training Methods

Use Google Cloud.

Note: Google Cloud isn't free. You need to pay depending on how long you use run the VM and what resources you use.

1. Create a Virtual Machine (VM) using a pre-configured Deep Learning VM Image.

export IMAGE_FAMILY="tf-latest-cpu"
export ZONE="us-west1-b" # Or any other required region
export INSTANCE_NAME="model-trainer"
export INSTANCE_TYPE="n1-standard-8" # or any other instance type
gcloud compute instances create $INSTANCE_NAME \
        --zone=$ZONE \
        --image-family=$IMAGE_FAMILY \
        --image-project=deeplearning-platform-release \
        --machine-type=$INSTANCE_TYPE \
        --boot-disk-size=120GB \
        --min-cpu-platform=Intel\ Skylake

2. As soon as instance has been created you can SSH to it:

gcloud compute ssh "jupyter@${INSTANCE_NAME}"

3. Train a model by following the instructions in the train_micro_speech_model.ipynb jupyter notebook.

4. Finally, don't forget to remove the instance when training is done:

gcloud compute instances delete "${INSTANCE_NAME}" --zone="${ZONE}"