Oído v2: real-time speech recognition on one ESP32-S3 core
¡Oído! is Spanish kitchen slang for heard, got it.
The real-time version of Oído. Version 1 transcribed at 1.8 to 2.0 times slower than speech on the chip; these two English models run at 0.54× (B) and 0.72× (A) the duration of the audio on one 240 MHz core of an ESP32-S3 (8 MB PSRAM, 16 MB flash, no neural accelerator), with the language model on, and the final text appears 259 ms (B) or 431 ms (A) after the end of speech when the audio arrives at the speed of speech. Measured on a physical board; the board's transcripts equal the host build's on 18 of 18 test clips.
Try it in your browser (no hardware, nothing leaves your device): Oído v2 demo, the same C engine compiled to WebAssembly.
Paper: Oído: Open-Vocabulary Speech Recognition on a $5 Microcontroller without a Neural Accelerator The version of the paper that describes these models (v2) is docs/oido-paper.pdf in the repository.
Code, firmware, tools, raw board measurements: github.com/lokutor-ai/oido (GPLv3, commercial licenses available).
A (oido_v2a.tnm) |
B (oido_v2b.tnm) |
|
|---|---|---|
| Conformer blocks | 16 | 12 |
| Weights | int4, 7.7 MB | int4, 5.9 MB |
| Streaming real-time factor on the chip, with the language model | 0.72 (beam 4, top 6) / 0.70 (beam 2, top 3) | 0.54 (beam 2, top 3) |
| Final text after the end of speech (audio at real-time speed) | 431 ms (281-574) | 259 ms (159-391) |
| LibriSpeech test-clean / test-other, streaming 0.64 s chunks, greedy | 10.01 / 19.90 | 13.43 / 26.10 |
| ... plus the language model | 7.14 / 16.37 (beam 4, top 6) | 10.63 / 22.42 (beam 2, top 3) |
| ... streaming 1.28 s chunks, plus the language model | 5.96 / 14.03 | 8.08 / 18.45 |
| ... full context, plus the language model | 4.58 / 10.71 | 6.15 / 14.41 |
Word error rates (%) come from the host build of the firmware engine, which runs the same C code with the same integer arithmetic and the same deterministic math as the chip. Streaming costs accuracy: with 0.64 s chunks the error rate on noisy "other" speech is above 10 %, and variant B trades about three points for a 25 % faster chip. For comparison, version 1 streaming (1.28 s chunks, language model, no speed-up) had 4.9 / 11.0 and ran at 1.8× real time on the chip.
What changed since version 1
The model was not the problem; memory traffic was. A streaming recognizer re-reads every weight for every chunk, and the chip reads 3.8 GB/s from internal SRAM, 89 MB/s from PSRAM and 32 MB/s from flash. So we (1) stack pairs of frames (80 ms encoder frames), (2) use a 64-channel convolutional front end instead of 176, (3) drop four blocks (B), (4) quantize the weights to int4 so that most of the model sits in PSRAM, and (5) overlap the weight stream with computation using the ROM's asynchronous data-cache preload, keep work buffers in SRAM, and make the decoder cheaper (table-driven exp/tanh, blank-frame shortcut, beam 2 / top-k 3 option). Section 6 of the paper has the ablation, the bandwidth measurements and the step-by-step table.
Use
git clone https://github.com/lokutor-ai/oido && cd oido
hf download lokutor-ai/oido-v2-realtime-int4 --local-dir models/v2 # oido_v2a.tnm, oido_v2b.tnm, oido_v2.tlm, tokenizer.model, config.json
cd esp32/host && make && python live_demo.py --model models/v2/oido_v2a.tnm # laptop microphone, same engine as the chip
esp32/tools/flash.sh /dev/ttyACM0 models/v2/oido_v2a.tnm models/v2/oido_v2.tlm # ESP32-S3-DevKitC-1 N16R8 + INMP441 microphone
Files
oido_v2a.tnm: variant A, int4 (TNM1 format). Header: 16 blocks, 64-channel front end, frame stacking 2, default chunk 8 and left context 64 encoder frames.oido_v2b.tnm: variant B (12 blocks), same format.oido_v2.tlm: the 1.3 M-parameter GRU language model (public-domain books only), with the weights tuned for these models (0.30, token bonus 1.0) stored in its header.tokenizer.model: the SentencePiece tokenizer of the original NVIDIA model (1024 pieces).
Limitations
- Test-other is above 10 % in every streaming mode; for noisy speech use longer chunks (
CONFIG_TASR_STREAM_CHUNK=16: about 20 % fewer errors, 0.6 s more delay) or the full-context mode of the same file. - English only. The firmware still waits 0.8 s of silence (configurable) before it declares the end of speech; the figures above start when that wait is over.
- Speed was measured with audio stored in flash and the weights partly in flash (the language model, the streaming state and the weights compete for the 8 MB of PSRAM); the live-microphone mode needs 0.9 MB more of PSRAM, see the repository for the measured effect.
- Two-core mode is still off: it gives wrong output on our boards (see the repository).
Training and license
Fine-tuned from NVIDIA's stt_en_conformer_ctc_small (CC-BY-4.0) with chunked attention (chunks of 0.32 to 1.28 s), noise/music/babble/reverberation augmentation, a light front end initialised by regression to the original one,
and 12k steps of int4 quantization-aware training, on about 1,900 hours of LibriSpeech, Common Voice 17 English, Multilingual LibriSpeech English, People's Speech (transcripts regenerated by parakeet-tdt-0.6b-v2),
AMI, VCTK, VoxPopuli English and OpenSLR 70/83. Because the data include share-alike corpora, the models are released under CC-BY-SA-4.0. The engine and firmware are GPLv3.
- Downloads last month
- 130
Model tree for lokutor-ai/oido-v2-realtime-int4
Base model
nvidia/stt_en_conformer_ctc_smallDatasets used to train lokutor-ai/oido-v2-realtime-int4
MLCommons/peoples_speech
facebook/multilingual_librispeech
Space using lokutor-ai/oido-v2-realtime-int4 1
Collection including lokutor-ai/oido-v2-realtime-int4
Paper for lokutor-ai/oido-v2-realtime-int4
Evaluation results
- WER (variant A, on-chip int4/int8 arithmetic, streaming 0.64 s chunks, language model) on LibriSpeech (clean)test set self-reported7.140
- WER (variant A, on-chip int4/int8 arithmetic, streaming 0.64 s chunks, language model) on LibriSpeech (other)test set self-reported16.370