tpl-opt-spot-vad-lvcsr tnl¶

This template optionally runs the wake word in slot 0 until it detects, then segments the audio following the wake word with a VAD and sends the segmented audio to the LVCSR or STT recognizer in slot 1.

slot controls whether tpl-opt-spot-vad-lvcsr waits for the wake word:

With slot == 0 it waits for the wake word before starting the VAD. In this mode the behavior is that of tpl-spot-vad-lvcsr.
With slot == 1 starts the VAD immediately and the behavior is that of tpl-vad-lvcsr.

You can change slot at runtime. Use this to gate only the first of a series of commands with a wake word.

tpl-opt-spot-vad-lvcsr has task-type==phrasespot.

Expected task types:

Slot 0: phrasespot
Slot 1: lvcsr

tpl-opt-spot-vad-lvcsr-1.28.0.snsr, tpl-spot-vad-lvcsr, tpl-vad-lvcsr

Operation¶

flowchart TD
  start((start))
  slotCheck0{slot == 0?}

  start --> slotCheck0
  slotCheck0 -->|yes| startWW
  slotCheck0 -->|no| fetch0

  subgraph slot0[<b>slot 0</b> &lpar;phrasespot&rpar;]
    startWW((start))
    fetchWW[/samples from ->audio-pcm/]
    audioWW(^sample-count)
    processWW[process]
    result(0.^result)
    stopWW((stop))
    startWW --> fetchWW
    fetchWW --> audioWW
    audioWW --> processWW
    processWW --> fetchWW
    processWW -->|recognize| result
    result --> stopWW
  end

  subgraph slot1[<b>slot 1</b> &lpar;lvcsr&rpar;]
    startSTT((start))
    startSTTfinal((start))
    stopSTT((stop))
    stopSTTpartial((stop))
    processSTT[process]
    partialSTT(^result-partial)
    intentSTT(^nlu-intent)
    slotSTT(^nlu-slot)
    resultSTT(^result)
    nluSTT{NLU<br>match?}

    slmSTT{SLM<br>included?}
    generateSTT[generate]
    slmstartSTT(^slm-start)
    slmresultpartialSTT(^slm-result-partial)
    slmresultSTT(^slm-result)

    startSTT --> processSTT
    processSTT ---->|hypothesis| partialSTT
    partialSTT --> stopSTTpartial

    startSTTfinal --> nluSTT
    nluSTT -->|yes| intentSTT
    nluSTT -->|no| resultSTT
    intentSTT --> slotSTT
    slotSTT --> resultSTT
    slotSTT -->|more| intentSTT

    resultSTT --> slmSTT
    slmSTT -->|yes| slmstartSTT
    slmSTT -->|no| stopSTT
    slmstartSTT -->|OK| generateSTT
    slmstartSTT -->|STOP| stopSTT
    generateSTT -->|response| slmresultpartialSTT
    slmresultpartialSTT --> generateSTT
    generateSTT -->|done| slmresultSTT
    slmresultSTT --> stopSTT
  end

  listenBegin(^listen-begin)
  listenEnd(^listen-end)

  stopWW --> listenBegin
  listenBegin --> fetch0

  fetch0[/samples from ->audio-pcm/]
  fetch1[/samples from ->audio-pcm/]
  audio0(^sample-count)
  audio1(^sample-count)

  silence(^silence)
  begin(^begin)
  END(^end)
  limit(^limit)

  process0[VAD process]
  process1[VAD process]

  final@{ shape: f-circ }

  slotCheck1{slot == 0?}

  fetch0 --> audio0
  audio0 --> process0
  process0 --> fetch0
  process0 -->|speech start| begin
  process0 -->|timeout| silence
  silence ~~~ final
  silence --> slotCheck1

  begin --> fetch1
  fetch1 --> audio1
  audio1 --> process1

  process1 --> startSTT
  stopSTTpartial --> fetch1

  process1 -->|speech end| END
  process1 -->|speech limit| limit
  END --> final
  limit --> final

  final --> startSTTfinal
  stopSTT --> slotCheck1

  slotCheck1 -->|no| fetch0
  slotCheck1 -->|yes| listenEnd
  listenEnd --> startWW

Operation flow.

Read audio data from ->audio-pcm.
Invoke ^sample-count.
If processing does not detect a wake word, continue at step 1.
Invoke 0.^result for the wake word.
Invoke ^listen-begin and start VAD processing.
Read audio data from ->audio-pcm.
Invoke ^sample-count.
If VAD processing does not detect the start of speech within the leading-silence timeout, invoke ^silence and continue at step 15.
Invoke ^begin if processing detects the start of speech, else continue at step 6.
Read audio date from ->audio-pcm.
Invoke ^sample-count.
If VAD processing detects an endpoint invoke either ^limit or ^end and continue at step 14.
Process VAD segmented audio in the LVCSR or STT recognizer
- Invoke ^result-partial with interim recognition result hypothesis.
- Continue at step 10.
Produce a final LVCSR or STT recognition hypothesis.
- Invoke ^nlu-intent and ^nlu-slot for each NLU intent found.
- Invoke ^result with the final recognition hypothesis.
- If there's no SLM, continue at step 15.
- Invoke ^slm-start, if the callback returns STOP, continue at step 15.
- Generate SLM result, invoking ^slm-result-partial on each generated token.
- Invoke ^slm-result with complete SLM result.
Invoke ^listen-end and start listening for the wake word again at step 1.

Register callback handlers with setHandler only for those events you're interested in.

Settings¶

^begin, ^end, ^limit, ^listen-begin, ^listen-end, ^nlu-intent, ^nlu-slot, ^result, ^result-partial, ^sample-count, ^silence, ^slm-result, ^slm-result-partial, ^slm-start

operating-point-iterator, vocab-iterator

audio-stream, audio-stream-first, audio-stream-last

->audio-pcm, audio-stream-from, audio-stream-to

audio-stream-size, audio-stream-size, backlog-interval, backoff, custom-vocab, delay, duration-ms, hold-over, include-leading-silence, include-wake-word-audio, leading-silence, low-fr-operating-point, max-recording, operating-point, partial-result-interval, samples-per-second, slot, stt-profile, sv-threshold, wake-word-at-end

lvcsr, phrasespot

live-spot.c, snsr-eval.c, PhraseSpot.java

Notes¶

Use this template for command and control type applications where commands are initiated with a wake word in certain contexts and not in others.

Set slot= 1 in the ^result handler, and slot= 0 in the ^silence handler. With this configuration the recognizer requires a wake word to start listening only for the first in a series of interactions. After this it will revert to requiring a wake word only if the user does not say anything for at least leading-silence ms.

VAD settings backoff, hold-over, leading-silence, max-recording, and trailing-silence apply to both slot 0 and slot 1, but include-leading-silence applies only to slot 0.

Set include-wake-word-audio= 1 to include the wake word audio in the samples passed to the LVCSR or STT recognizer. STT hypotheses do not include the wake word text unless Sensory specifically configured the model to do so.

The ^result-partial and ^result events are for the LVCSR or STT recognizer in slot 1. If you need direct access to the wake word result, prefix the event with the slot path: 0.^result Use the slot prefix to read values in the 0.^result event handler too, for example call getString with key 0.text to read the wake word transcription.

Examples¶

Select wake-word or VAD-only behavior¶

% cd $HOME/Sensory/TrulyNaturalSDK/7.9.0-pre.1

% bin/snsr-edit -o opt-vg-stt.snsr\
    -t model/tpl-opt-spot-vad-lvcsr-1.28.0.snsr\
    -f 0 model/spot-voicegenie-enUS-6.5.1-m.snsr\
    -f 1 model/stt-enUS-general-medium-2.4.4-pnc.snsr\
    -s include-wake-word-audio=1

# Say "Voice genie, open the sunroof."
% snsr-eval -vt opt-vg-stt.snsr
Using live audio from default capture device. ^C to stop.
P   2090   2530 (0.9433) Open. This
P   2130   3010 (0.8589) Open the sun rois
  1050   3435 [^end] VAD speech region.
  2130   3130 (0.9916) Open the sunroof.
^C

# Select the VAD-only path with slot=1
# Say "Close all the windows"
% snsr-eval -vt opt-vg-stt.snsr -s slot=1
Using live audio from default capture device. ^C to stop.
P   3000   3320 (0.7413) Close
P   3040   3760 (0.8003) Close. All the w
P   3040   4240 (0.9996) Close all the windows
  2685   4740 [^end] VAD speech region.
  3040   4360 (0.9999) Close all the windows.
^C

Use trailing wake word 7.7.0¶

Recognize a phrase with the wake word at either end of an utterance.

% cd $HOME/Sensory/TrulyNaturalSDK/7.9.0-pre.1

% bin/snsr-edit -o opt-vg-stt-vg.snsr\
    -t model/tpl-opt-spot-vad-lvcsr-1.28.0.snsr\
    -f 0 model/spot-voicegenie-enUS-6.5.1-m.snsr\
    -f 1 model/stt-enUS-general-medium-2.4.4-pnc.snsr\
    -s include-wake-word-audio=1\
    -s wake-word-at-end=1

# Say "Voice genie, set the radio to 91.5 FM."
% bin/snsr-eval -vt opt-vg-stt-vg.snsr
Using live audio from default capture device. ^C to stop.
P   4790   4950 (0.7039) Sid
P   4830   5470 (0.9663) Set. The radio
P   4870   5630 (0.5832) Set. The radio. T
P   4910   6310 (0.9998) Set the radio to ninety one
P   4870   6670 (0.9989) Set the radio to ninety one point
P   4870   6990 (0.9971) Set the radio to ninety one point. Five
P   4910   7350 (0.9159) Set the radio to ninety one point. Five af
  3510   7965 [^end] VAD speech region.
  4870   7630 (0.9999) Set the radio to ninety one point. Five F. M.

# Say "Will it rain in Portland tomorrow, Voice Genie?"
 15090  17160 [^end] VAD speech region.
 15410  17050 (0.9936) Will it rain in Portland tomorrow?
^C