Custom on-device voice commands from zero recordings

Embedded hardware applications—such as camera control (flip_camera), HVAC systems (set_eco_mode), or audio headsets (mute_mic)—require targeted spoken command recognition without full automatic speech recognition (ASR) pipelines. Traditional custom voice recognition pipelines require collecting multi-speaker audio recordings, training an acoustic model, and compiling the model to fit edge chip power and memory budgets.

This post details a zero-recording voice command recognition architecture. By combining synthetic text-to-speech (TTS) generation, heavy acoustic data augmentation, a frozen acoustic encoder, and a lightweight classification head, custom voice command models can be trained from text specifications alone and compiled into self-contained 8.58 MB INT8 artifacts for Qualcomm edge silicon.

8.58 MB

self-contained bundle

int8 encoder + numpy head, ~5.6 MB zipped

220

head-only training steps

seconds, not the GPU-hours a fine-tune costs

bytes to a server at inference

no PyTorch, no transformers

a camera command model · input to output

input · spoken

“take a photo”
“start recording”
“switch to the front camera”
“what's the weather like”

on-device model8.58 MB · frozen encoder + head

output · action

take_photo0.98

start_video0.96

flip_camera0.95

unknown

Illustration of the input-to-output contract. Confidence values are illustrative, not measured.

Edge Architecture Constraints vs. Cloud Processing

Cloud-based automatic speech recognition is impractical for embedded product hardware across four distinct engineering constraints:

Privacy: Transmitting raw user audio off-device introduces compliance and security risks.
Latency: Network round-trip delays introduce non-deterministic execution spikes.
Availability: Offline environments break cloud-dependent voice interfaces entirely.
Power and Bandwidth Costs: Streaming uncompressed audio over wireless links (BLE/Wi-Fi) drains battery reserves faster than local NPU execution.

Consequently, edge systems require local, self-contained recognition pipelines optimized for low memory footprints and deterministic execution latency.

Architectural Comparison: Cascade vs. Direct Intent Classifier

Conventional voice command recognition relies on a cascaded architecture: an acoustic model transcribes audio to text, and a natural language processing (NLP) model extracts command intents from the generated text.

Conventional cascaded architecture: ASR encoder generates feature vectors, ASR decoder generates text tokens autoregressively, and an LLM classifies intent (~1.1 s on Apple M-series hardware).

audio

whisper encoderlog-mel → features

whisper decoderautoregressive → text

functiongemma270M LLM · reads the text

intent

Cascaded systems introduce substantial latency and memory overhead by converting intermediate audio representations into text before classifying intent. In contrast, Spoken-Language Understanding (SLU) maps audio representations directly to intent classifications.

By replacing the autoregressive decoder and language model with a 2-layer classifier on top of a frozen acoustic embedding, execution latency drops from ~1.1 seconds down to 9.6 milliseconds while eliminating autoregressive decoding memory overhead.

Architecture comparison: Cascaded ASR+LLM vs Direct SLU Classifier

Cascaded ASR + LLM~1.1 s

audio

whisper encoderaudio → features

decode → text → 270M LLMtranscribe, then parse

intent

Direct SLU Classifier9.6 ms

audio

moonshine encoderfrozen, shared

2-layer headtrained per client

intent

Direct SLU maps acoustic embeddings straight to command classes, bypassing text token generation and reducing latency by 120×.

Frozen Speech Encoders and Minimal Classification Heads

Extracting acoustic features from raw audio is a domain-agnostic task. We use a shared pre-trained speech encoder (UsefulSensors/moonshine-tiny) with frozen weights, training only a lightweight two-layer MLP classification head per application.

Because encoder parameters remain constant, training does not require backpropagating through the acoustic backbone. Instead, acoustic embeddings are pre-computed and cached in RAM during dataset synthesis.

Computational complexity of 220 full-batch optimization steps

Caching pre-computed embeddings transforms model training from acoustic backpropagation into lightweight tensor matrix multiplication over a small matrix ( $M \times d$ ). 220 full-batch steps complete in ~2.5 seconds on standard CPU hardware without GPU acceleration.

Data flow during training: Synthetic audio passes through the frozen encoder once, and the classification head optimizes over cached embeddings.

typed phrases

synthetic voices5 TTS voices + prosody

shared encoderfrozen

cached embedding

2-layer headtrained per client

label incl. unknown

shared + constant
trained per client

Synthetic Audio Generation and Augmentation Pipeline

To eliminate manual audio recording, training datasets are generated purely from text specifications using multi-stage synthesis and augmentation:

recordings

every clip synthesized from typed text

5 / 2

TTS voices, train / held-out

accents kept strictly separate

generic negatives

train the unknown class

Data augmentation stack parameters (`backend/trainer.py`)

Stage	Augmentation Vectors	Parameter Ranges	Application Frequency
Text Level	Synonym maps, carrier templates, verb bridging	10 prefix templates / 5 suffix templates	Per input phrase
Voice Level	TTS accents, rate jitter, pitch shift	Speaking rate −12% to +12%, pitch ±15 Hz	Per synthetic sample
Audio Level	Speed perturbation, synthetic impulse response, gain, SNR noise	Speed 0.9–1.1×, IR 30–120 ms (p=0.4), gain −6 to +3 dB, SNR 5–25 dB (p=0.85)	Per audio waveform
Feature Level	Gaussian noise injection, Mixup regularization	Gaussian std = 0.1 × dim_std, Mixup Beta(0.4, 0.4)	Directly on cached embeddings

Feature-level augmentation operates on cached embeddings without incurring additional TTS or audio processing overhead.

Text Paraphrasing: Carrier templates (“can you”, “please”) and domain-constrained synonym mappings expand base phrases without introducing cross-domain semantic drift.
Multi-Accent TTS Synthesis: Text inputs are synthesized across 5 distinct TTS voices (US, UK, Indian, Australian accents) with pitch and rate variations.
Acoustic Signal Perturbation: Audio waveforms undergo speed perturbation (0.9–1.1×), synthetic room impulse response (RIR) convolution (30–120 ms decay), gain adjustment (−6 to +3 dB), and additive background noise (5–25 dB SNR).

Constraining synonym expansion rules

Unconstrained LLM paraphrasing can introduce invalid phrases (e.g. “film a photo” or “screenshot the stream”). Domain-specific verb bridging restricts synonym substitutions strictly within semantic boundaries (e.g. substituting “record” with “capture” for video targets only).

Synthetic room impulse response (RIR) generation

To simulate acoustic room reflections without recording physical environments, waveforms are convolved with synthetic exponentially decaying noise bursts ( $T_{60} \in [30, 120]$ ms). This forces the classifier to learn features invariant to reverberation.

Latent Embedding Space Augmentation

Because the acoustic encoder is frozen, audio waveforms are passed through the backbone once to produce cached embedding matrices. We apply additional augmentation directly in embedding space:

Gaussian Noise Injection: Zero-mean Gaussian noise ( $\sigma_j = 0.1 \times \text{std}(E_{i, j})$ ) is added to cached feature vectors.
Mixup Regularization: Convex combinations of embedding vectors and target labels are generated using $\lambda \sim \text{Beta}(0.4, 0.4)$ .

Embedding augmentation regularizes decision boundaries and increases model robustness against feature shifts introduced during quantization.

Interaction between embedding noise and INT8 quantization

Injecting noise into embedding space during training teaches the classification head to tolerate small variations in feature vector coordinates. This variance tolerance matches the magnitude of rounding errors introduced during INT8 model quantization.

Synthetic sample count generated from 3 seed phrases

Base text expansion42 synthetic clips

Full text + audio + feature augmentation360 augmented embeddings · 100% held-out voice split

Offline INT8 Quantization and Feature Drift Verification

The acoustic encoder (UsefulSensors/moonshine-tiny) is quantized to INT8 precision offline using ONNX Runtime quantization utilities (backend/build_encoder.py), producing a fixed 8.58 MB binary artifact.

To ensure INT8 quantization does not degrade feature representation, quantized outputs are validated against FP32 reference vectors using cosine similarity across test audio samples. A similarity threshold of $\cos(\theta) \ge 0.99$ is enforced as a deployment gate.

Mathematical resilience of pooled embeddings to quantization error

Because the classification head operates on mean-pooled embedding directions rather than raw high-frequency activations, small quantization perturbations ( $\cos(\theta) \ge 0.99$ ) exert minimal impact on linear decision boundaries.

Out-of-Vocabulary Handling via Explicit Unknown Class

In deployment, speech input frequently includes out-of-vocabulary phrases or background conversation. Softmax probability thresholding (max_prob < threshold) often fails when out-of-vocabulary audio projects strongly onto a single target class.

To address this, the pipeline includes an explicit unknown class ( $N+1$ ). During dataset generation, 12 generic non-command phrases (“what time is it”, “tell me a joke”, “never mind”) are synthesized and assigned to the unknown target index, establishing an explicit rejection boundary in embedding space.

Probability thresholding vs. explicit negative classification

Thresholding assumes out-of-vocabulary inputs produce uniform softmax distributions. However, out-of-vocabulary audio can produce localized high-confidence activations. Explicitly training an unknown class establishes decision boundaries around out-of-domain feature clusters.

Empirical Evaluation and Held-Out Generalization

Model accuracy was evaluated on synthetic test sets using held-out TTS speaker voices (en-US-AriaNeural, en-GB-RyanNeural) and unseen phrase variations at neutral prosody.

Classification accuracy on held-out TTS voices (before vs after verb-bridging augmentation)

Voice	Predicted Intent	Confidence	Evaluation Status
Samantha	unknown	0.41	False Negative (target: start_video)
Daniel	unmute	0.96	False Positive (unrelated intent)
Karen	start_video_recording	0.48	Correct Classification

Adding domain-constrained verb-bridging eliminated false positive classifications and increased held-out intent accuracy from 98.2% to 99.3%.

Updating augmentation rules to include media-specific verb bridging raised held-out intent accuracy from 98.2% to 99.3% and exact-match action accuracy from 97.7% to 99.1%.

CPU Compute Budget and Generation Constraints

When deployed on CPU-only infrastructure (e2-standard-4 instance), text expansion parameters are set to EXPAND_N=3 (compared to EXPAND_N=4 on GPU instances) to constrain total dataset generation and model compilation times under 60 seconds.

Caching pre-synthesized voice audio embeddings across common command templates further reduces CPU build latency.

Deployment Artifacts and System Dependencies

The compiled deployment bundle is fully self-contained, requiring only onnxruntime, numpy, scipy, and soundfile runtime dependencies. PyTorch and Hugging Face Transformers are excluded from runtime builds.

The total package size comprises:

Quantized Encoder (moonshine_int8.onnx): 8,582,074 bytes (~8.58 MB).
Classification Head & Metadata (head.npz): ~45 KB NumPy array archive.
Compressed Archive Size: ~5.6 MB zip.

Snapdragon NPU compilation target details

The profiled benchmark reflects FP32 ONNX graph execution on Snapdragon 8 Elite Hexagon hardware. Compiling INT8 ONNX graphs directly to QNN/DLC NPU context binaries represents the immediate deployment target.

Execution Speed and Power Consumption Benchmarks

End-to-end inference latency was benchmarked on Apple M-series hardware against a standard Faster-Whisper + FunctionGemma (270M) ASR-LLM cascade:

120×

faster than cloud cascade

1.1 s → 9.6 ms per command

180×

less energy per command

1.5× lower power × 120× less time

1,000×

fewer bytes transferred

50-byte payload vs streaming audio

Under macOS powermetrics profiling, the Direct SLU classifier consumed 7.8 W over 9.6 ms (0.075 Joules/inference), while the cascaded ASR-LLM pipeline consumed 11.8 W over 1,100 ms (12.98 Joules/inference), demonstrating a ~180× reduction in total energy per command execution.

Additionally, local intent classification replaces raw audio streaming over Bluetooth Low Energy (BLE) with a 50-byte JSON intent payload, reducing wireless transmission overhead by over 1,000×.

Future Work

Quantized NPU Compilation: Compile INT8 ONNX encoder graphs directly to Qualcomm QNN context binaries for earbud-class NPUs.
Real-World Acoustic Benchmarking: Expand evaluation datasets to include physical microphone array recordings in reverberant and high-noise environments.
Multilingual Synthetic Pipeline: Extend text augmentation and TTS voice sets to support non-English voice command classification.

Conclusion

Combining frozen acoustic encoders, synthetic TTS data generation, and embedding-space augmentation enables lightweight, on-device voice command recognition without manual audio collection. By eliminating autoregressive transcription decoders, inference latency drops to 1.24 ms on Snapdragon hardware while maintaining a 8.58 MB footprint suitable for edge deployment.

Glossary

Glossary — every term, defined

Spoken-language understanding (SLU): Mapping speech straight to an intent, skipping the transcription step a speech-to-text-plus-parser cascade uses.
Intent: A command you name and want recognized, such as capture_photo or mute. You choose the full set.
Unknown: The extra class returned when a phrase matches none of your intents, so an app can ignore stray speech or re-prompt instead of guessing.
Speech encoder: A model that turns raw audio into a vector capturing what was said, largely independent of the speaker. Here it is fixed and shared, not trained per project.
Moonshine-tiny: The open speech encoder used (UsefulSensors/moonshine-tiny), run frozen as a feature extractor.
Embedding: The fixed-length vector the encoder produces for a clip. Similar-sounding commands land near each other, which is what the classifier reads.
Frozen weights: Weights held constant during a build. The encoder is frozen; only the head learns.
Head (classifier): The small two-layer network trained per project. It reads an embedding and outputs one score per label, and it is the only part that learns your commands.
Text-to-speech (edge-tts): Turning written phrases into spoken audio. The build uses Microsoft edge-tts across several voices, so no recordings are needed.
Mixup: Training on blends of two examples and their labels, which smooths the classifier boundaries.
int8 / quantization: Storing a model in 8-bit numbers to shrink and speed it up. The encoder ships this way at cosine >0.99 to full precision.
Held-out voices: Speakers used only to measure a model, never to build it, so reported accuracy reflects voices it has not seen.
NPU: Neural processing unit. Low-power silicon built to run models, where a small model like this one belongs.
Qualcomm AI Hub: Qualcomm's service for compiling and profiling a model on real Snapdragon hardware. Source of the ~1.24 ms figure.