AudioQueue and ScreenCaptureKit capture, 20 ms PCM input, bounded model workers and SQLite-backed retrieval.
Private native application. Device-dependent capture and dictation require separate validation.
Technical deep dive Architecture, implementation & tradeoffs
Capture keeps running while models decode.
- 01
Capture
AudioQueue · 20 ms PCMA native 16 kHz audio path captures and buffers continuously.
- 02
Separate
screen audio ≠ microphoneScreenCaptureKit and microphone capture retain independent stream paths.
- 03
Decode
bounded workers + GPU gateProvisional transcription updates quickly; a separate final pass revisits the recording.
- 04
Remember
lossless chunks → SQLitePersist recordings and local memory for later reconstruction.
Capture → durable audio → provisional text → final decode. Transcript revision does not require losing or recapturing the original signal.
- Concurrency
- Audio callbacks are separate from inference work
- Scheduling
- Bounded local workers with serialised GPU work
- Recovery
- Retained audio supports independent final decoding
Capture and inference lifecycle
- Implemented separate paths for AudioQueue dictation and ScreenCaptureKit system-audio plus microphone capture.
- Used 20 ms PCM callbacks and rolling lossless 16 kHz mono audio files for dictation.
- Built provisional phrase decoding followed by an independent quality pass over the original audio.
Resource and failure handling
- Bound the number of loaded model workers and serialized GPU inference requests.
- Kept microphone startup independent of remote-backend discovery and model loading.
- Preserved provisional text when finalisation fails, with explicit microphone-open, first-audio, first-text and stop-to-final timings.
Personal memory
- Built local multilingual transcription and speaker-aware processing around existing speech models.
- Added SQLite-backed memory, retrieval, correction handling and optional backend-routing contracts.
- Made capture, transcription, storage and action boundaries visible in the native application.
Latency is a sequence of boundaries.
The time before useful text appears includes opening the input device, receiving the first real samples, accumulating context and scheduling inference. I separated those stages so a slow device open does not disappear inside a single transcription-time number.
Actively developed native application. Hardware-dependent capture and cross-application actions have their own device-validation requirements.