Voice

NeuralCleave includes a fully local voice pipeline: Whisper for speech-to-text, a 3-tier TTS engine (ElevenLabs → Kokoro → pyttsx3), OpenWakeWord for always-on wake word detection, and optional voice cloning.

Pipeline overview

text

Microphone
    │
    ▼
OpenWakeWord (continuous listener, optional)
    │ wake word detected
    ▼
Whisper STT ──── transcribed text ────▶ AgentRuntime
                                              │
                                         LLM response
                                              │
                                              ▼
                                        TTS Engine
                                     (ElevenLabs / Kokoro / pyttsx3)
                                              │
                                              ▼
                                         Speaker output

Enabling voice

bash

# Start the gateway with voice enabled
neuralcleave voice start

# Or set in config.toml and use the normal start command
neuralcleave start

toml

[voice]
enabled = true
stt_model = "base"
tts_provider = "kokoro"

ℹ

Voice requires portaudio on the host. Install it with brew install portaudio (macOS), apt install portaudio19-dev (Linux), or the Windows binary from portaudio.com. The Python packages sounddevice and soundfile are installed automatically with pip install neuralcleave[voice].

Speech-to-text (Whisper)

NeuralCleave uses faster-whisper (CTranslate2 backend) for low-latency local transcription. No audio leaves the device.

Model	VRAM	Speed	Accuracy
`tiny`	~75 MB	Fastest	Good for simple commands
`base`	~145 MB	Fast	Good all-around (default)
`small`	~466 MB	Moderate	Better accents
`medium`	~1.5 GB	Slower	Near-human accuracy
`large-v3`	~3.1 GB	Slow	Best in class

toml

[voice]
stt_model  = "base"     # tiny | base | small | medium | large-v3
stt_device = "cpu"      # cpu | cuda  (cuda requires NVIDIA + cuDNN)

Text-to-speech (3-tier)

TTS falls back through three engines in order of preference. Configure tts_provider to select your preferred tier:

Tier	Provider	Quality	Cost	Latency
1	ElevenLabs	Best — natural prosody, emotion	Paid (API)	~400 ms
2	Kokoro	Excellent — local neural TTS	Free, local	~100 ms
3	pyttsx3	Adequate — system speech engine	Free, local	~20 ms

toml

[voice]
tts_provider        = "kokoro"               # elevenlabs | kokoro | pyttsx3
elevenlabs_api_key  = "ENV:ELEVENLABS_API_KEY"
elevenlabs_voice_id = ""                     # leave blank to use account default

Wake word (OpenWakeWord)

When wake_word_enabled = true, the gateway listens continuously in the background. Audio is processed locally on CPU; no audio is sent anywhere. The wake word triggers Whisper to begin recording and transcribing.

toml

[voice]
wake_word_enabled = true
wake_word_model   = "hey_jarvis"  # built-in model name, or path to a .tflite file

Built-in models: hey_jarvis, alexa, hey_mycroft, hey_rhasspy. Custom .tflite models trained with the OpenWakeWord trainer are also supported by pointing to the file path.

Continuous listener

The continuous listener is the always-on mode that combines wake word detection with automatic end-of-speech detection. After the wake word fires, Whisper records until it detects a pause longer than 800 ms (configurable), then sends the transcription to the agent. The agent responds via TTS and the listener returns to wake-word standby.

bash

# Start the continuous voice listener
neuralcleave voice start

# Stop it
neuralcleave voice stop

# Check status
neuralcleave voice status

Voice cloning

Voice cloning requires an ElevenLabs Instant Voice Clone (IVC) subscription. Provide 30–60 seconds of clean reference audio:

bash

# Clone a voice from a WAV file
neuralcleave voice clone --name "MyVoice" --audio reference.wav

# Use the cloned voice ID in config
neuralcleave voice list-voices

The resulting voice ID can be set as elevenlabs_voice_id in the config, or overridden at runtime via the settings API:

bash

curl -X POST http://localhost:7432/api/v1/settings \
  -H "Content-Type: application/json" \
  -d '{"voice_id": "YOUR_VOICE_ID"}'

Voice dependencies

bash

# Install voice extras
pip install neuralcleave[voice]

# This installs: faster-whisper, kokoro-tts, pyttsx3, sounddevice,
# soundfile, openWakeWord, pyaudio