Keyword Spotting for iOS

Streaming multi-keyword spotter on the VoxRT custom on-device inference runtime. 636 K-parameter Conformer-Medium (d=96 × 4 blocks), 16 kHz mono in, per-class sigmoid posteriors out + threshold-crossing events. Ships with a 14-way vocabulary: yes / no / cancel / play / pause / next / previous / up / down / back / on / off / voxrt / hey_vox.

• Status: production-ready for arm64 iOS. Same runtime as the Linux + browser + Android SDKs; numeric parity vs PyTorch reference validated bit-identically across the NEON backend on the deployed model — see Performance.
• Current version: v0.1.0
• Minimum iOS: 16.0
• Architectures shipped: arm64 (iPhone / iPad, NEON-accelerated) + simulator slices (arm64 + x86_64)
• License: Apache-2.0 (Swift wrapper) · proprietary (compiled runtime, redistribution allowed via this Swift Package)
• Model weights: proprietary in-house (synthetic + open training data; no upstream license obligations)

What is VoxRT?

VoxRT is a from-scratch inference runtime for on-device speech models. No ONNX Runtime, no PyTorch Mobile, no LiteRT — a custom Rust core sized and tuned for streaming voice workloads on phone-class hardware.

VoxrtKws is the keyword-spotting product on that runtime, alongside VoxrtWakeWord (single-phrase always-on detection), VoxrtAsr (streaming ASR) and VoxrtSilero (VAD). All four share the same Rust runtime crate and the same NEON kernel set.

Custom keyword vocabularies (your own brand terms, additional languages, larger class counts) are part of the commercial VoxRT SDK tier. Contact [email protected].

Model

Streaming Conformer-Medium — d_model=96, 4 blocks, 8 attention heads, 636 K params, fp16, 1.28 MB .vxrt. 14 output classes at 25 fps (encoder stride = 4 × 10 ms). Firing rule: sigmoid ≥ threshold (default 0.9) sustained for consecutiveFramesRequired emits (default 3 = 120 ms), then per-class cooldown of cooldownFrames emits (default 25 = 1 s).

Model quality

Held-out speaker-disjoint test split, per-class threshold-swept macro-F1 on the 14-way vocabulary:

• F1 macro: 0.9671 at the default operating point (threshold 0.9, consecutive-3 firing, cooldown 25 emits).
• Per-class ROC AUC ≥ 0.99 across every keyword.
• Trained on the M13 recipe — capacity headroom preserves the noise-robustness of M10 while matching the M9 clean-set headroom (see internal docs/kws/M13-STATUS.md).

Default operating point tuned for false-fire suppression in an always-on setting; lower the threshold at runtime via setThreshold if your application can tolerate more triggers in exchange for higher recall.

Performance

arm64 device builds use NEON 4-wide FMA in every hot kernel (mel frontend, Conformer MHA, depthwise conv module, FFN, LayerNorm, softmax, sigmoid). All ARMv8-A Apple silicon mandates NEON, so no runtime detection is needed.

Live-mic bench = AVAudioEngine at 16 kHz mono i16 PCM, 100 ms chunks, single thread, sustained (post-warmup) session RTF. RTF = wall-time / audio-time (lower is better):

Both the iPhone 13 Pro Max and Snapdragon 662 rows are direct measurements — the runtime is the same libvoxrt_kws static archive linked into VoxrtKwsNative.xcframework. Other rows scale from the A15 baseline by clock + µarch — no per-device tuning.

At RTF ≈ 0.09 on an iPhone 13 Pro Max, the KWS engine burns ~9 % of one core during continuous listening — 11× headroom above realtime. That leaves budget for streaming ASR or wake-word running in parallel on the same audio stream.

Cross-backend numeric parity — the same libvoxrt_kws produces bit-identical output vs the PyTorch reference across every backend the runtime targets:

Post-diff sits ~7 × below the abs-tolerance floor of the golden CI (5 × 10⁻³). Detection scores and emit counts are identical across all three backends.

Binary footprint

• Swift wrapper source: ~10 KB total (one file)
• VoxrtKwsNative.xcframework.zip (downloaded by SPM): ~4 MB compressed (device + simulator slices)
• After SPM extraction + linker dead-code elimination on the device-only path: ~1 MB delta in your app binary
• KWS model voxrt_kws.vxrt: ~1.28 MB fp16 (downloaded separately)

Net effect on a consuming iOS app's IPA: roughly 2–3 MB once xcframework device slice + .vxrt + Swift wrapper are linked and bundled.

Install

In Xcode: File → Add Package Dependencies → paste:

https://github.com/VoxRT/voxrt-kws-ios

…and pin to v0.1.0.

Or in Package.swift:

dependencies: [
    .package(url: "https://github.com/VoxRT/voxrt-kws-ios.git", from: "0.1.0"),
],

Get the model

The model weights are NOT bundled with the package — fetch them once from voxrt-kws-models:

https://github.com/VoxRT/voxrt-kws-models/releases/download/v0.1.0/voxrt_kws.vxrt

SHA-256: 2fdddc5ea63b26342b28a9bd1c78bb946f0cc8943b4086c46e51c53ac6cc3353

Three common bundling patterns for a ~1.3 MB asset:

• Bundle in app resources — drag voxrt_kws.vxrt into your Xcode target and load with VoxrtKwsEngine(bundleResource: "voxrt_kws"). Works offline from first launch.
• Download on first run — URLSession fetch into FileManager.default.urls(for: .applicationSupportDirectory, ...), verify the SHA-256, then load with VoxrtKwsEngine(modelURL: cachedFile). Lets you swap models without an app update.
• App Thinning / On-Demand Resources — Apple's per-asset delivery if you want the App Store to host the file.

Download-on-first-run snippet

import CryptoKit

private let kKwsModelURL = URL(string:
    "https://github.com/VoxRT/voxrt-kws-models/releases/download/v0.1.0/voxrt_kws.vxrt"
)!
private let kKwsModelSHA256 = "2fdddc5ea63b26342b28a9bd1c78bb946f0cc8943b4086c46e51c53ac6cc3353"

func ensureKwsModel() async throws -> URL {
    let fm = FileManager.default
    let dir = try fm.url(
        for: .applicationSupportDirectory, in: .userDomainMask,
        appropriateFor: nil, create: true
    )
    let dest = dir.appendingPathComponent("voxrt_kws.vxrt")
    if fm.fileExists(atPath: dest.path),
       sha256Hex(try Data(contentsOf: dest)) == kKwsModelSHA256 {
        return dest
    }
    let (tmpURL, _) = try await URLSession.shared.download(from: kKwsModelURL)
    let bytes = try Data(contentsOf: tmpURL)
    guard sha256Hex(bytes) == kKwsModelSHA256 else {
        throw NSError(domain: "voxrt.kws", code: 1,
                      userInfo: [NSLocalizedDescriptionKey: "model SHA-256 mismatch"])
    }
    if fm.fileExists(atPath: dest.path) { try fm.removeItem(at: dest) }
    try fm.moveItem(at: tmpURL, to: dest)
    return dest
}

private func sha256Hex(_ d: Data) -> String {
    SHA256.hash(data: d).map { String(format: "%02x", $0) }.joined()
}

// Then, in your app:
let modelURL = try await ensureKwsModel()
let engine = try VoxrtKwsEngine(modelURL: modelURL)

Quick start

import VoxrtKws

// 1. Resolve the bundled model URL.
guard let modelURL = Bundle.main.url(forResource: "voxrt_kws",
                                     withExtension: "vxrt") else {
    fatalError("voxrt_kws.vxrt not found in bundle")
}

// 2. Build the engine.
let engine = try VoxrtKwsEngine(modelURL: modelURL)
// engine.classNames == ["yes", "no", "cancel", "play", …] (length 14)

// 3. Feed Int16 PCM (mono, 16 kHz) blocks of any size — 100 ms
//    blocks are the recommended pace for AVAudioEngine taps.
//    processPcm returns any threshold-crossings that occurred
//    during this push; usually empty.
for d in try engine.processPcm(pcmInt16Array) {
    print("'\(d.className)' @\(d.timestampSec)s score=\(d.score)")
}

processPcm / reset / close are synchronous and stateful. The engine does NOT own a worker thread. Drive it from your own capture thread.

Live microphone example

The canonical streaming pattern — capture-thread owns the AVAudioEngine tap, engine is a stateful function.

import AVFoundation
import VoxrtKws

let session = AVAudioSession.sharedInstance()
try session.setCategory(.record, mode: .measurement)
try session.setPreferredSampleRate(16_000)
try session.setActive(true)

let audioEngine = AVAudioEngine()
let input = audioEngine.inputNode
let hwFormat = input.outputFormat(forBus: 0)

let voxrtFormat = AVAudioFormat(
    commonFormat: .pcmFormatInt16,
    sampleRate: 16_000,
    channels: 1,
    interleaved: true
)!
let converter = AVAudioConverter(from: hwFormat, to: voxrtFormat)!

guard let modelURL = Bundle.main.url(forResource: "voxrt_kws",
                                     withExtension: "vxrt") else { fatalError() }
let kws = try VoxrtKwsEngine(modelURL: modelURL)

input.installTap(onBus: 0, bufferSize: 4_096, format: hwFormat) { hwBuf, _ in
    let outCap = AVAudioFrameCount(
        Double(hwBuf.frameLength) * 16_000 / hwBuf.format.sampleRate + 256
    )
    guard let outBuf = AVAudioPCMBuffer(pcmFormat: voxrtFormat, frameCapacity: outCap) else {
        return
    }
    var err: NSError?
    converter.convert(to: outBuf, error: &err) { _, status in
        status.pointee = .haveData
        return hwBuf
    }
    if err != nil { return }
    guard let i16Ptr = outBuf.int16ChannelData?[0] else { return }
    let samples = Array(UnsafeBufferPointer(start: i16Ptr, count: Int(outBuf.frameLength)))
    do {
        for d in try kws.processPcm(samples) {
            DispatchQueue.main.async {
                print("KWS: '\(d.className)' score=\(d.score)")
            }
        }
    } catch { /* surface to UI */ }
}

try audioEngine.start()

Permission: add NSMicrophoneUsageDescription to your Info.plist explaining why the app records audio, and call AVAudioApplication.requestRecordPermission(completionHandler:) (iOS 17+) or AVCaptureDevice.requestAccess(for: .audio) before starting AVAudioEngine.

Tuning

Threshold

Default is 0.9 (matches the browser + Linux + Android SDKs). Lower for higher recall, raise for stricter precision:

try engine.setThreshold(0.85)   // a bit more recall
try engine.setThreshold(0.95)   // stricter, but loses recall

Consecutive-frames-required

The class posterior must stay ≥ threshold for consecutive emits before firing. Default is 3 (= 120 ms at 25 fps). Higher = more robust against transient spikes, longer detection latency:

try engine.setConsecutiveFramesRequired(5)   // 200 ms

Cooldown

After a detection on a class, the engine suppresses further events on that same class for cooldownFrames emits. Default is 25 (= 1 s at 25 fps).

try engine.setCooldownFrames(50)   // 2 seconds

API

VoxrtKwsEngine

KwsDetection

public struct KwsDetection: Equatable {
    public let classIndex: Int      // 0-based index in classNames
    public let className: String    // resolved name (e.g. "play")
    public let timestampSec: Float  // seconds since engine start (or last reset)
    public let score: Float         // sigmoid score in [0, 1]
    public let emitIndex: UInt64    // 0-based encoder-frame index (25 fps)
}

VoxrtKws (namespace)

VoxrtKws.nativeVersion              // "0.1.0"
VoxrtKws.abiVersion                 // (major: UInt16, minor: UInt16)

License

• Swift wrapper source (this Swift Package): Apache-2.0. See LICENSE.
• Compiled runtime (VoxrtKwsNative.xcframework): proprietary, redistributable under the terms in LICENSE-BINARY.
• KWS model (voxrt_kws.vxrt): proprietary, distributed separately under the voxrt-kws-models license terms.

For commercial integration, custom vocabularies, or licensing terms beyond redistribution of the unmodified library, contact [email protected].

Synced from voxrt-kws-ios on 2026-07-22 — the GitHub repo is the source of truth. Found a mismatch? The repo wins.