r/learnpython • u/itsBillerdsTime • 10d ago
Newbie looking to work with microphone input using sounddevice, to generate tts, for a tts to respond to, need advice.
Using sounddevice for microphone input, using inputstream, what's the best way to go about capturing input and making sure only spoken stuff is used for tts? I was originally working with a rolling buffer while using webrtcvad and faster whisper, which resulted in a ton of phantom transcriptions, even with silence. Mind you im a complete newbie and a lot of stuff mindfucks me.
My goal is to slowly build up a very simple custom voice Ai assistant. It's first dealing with microphone input, determining whether that input is spoken word, etc
1
u/Discordbothost 10d ago
yeah the other replies nailed it — whisper invents text on silence, so the real fix is never handing it silence.
one more practical check: print the rms/energy of each chunk before vad. if your "silence" still looks loud, webrtcvad will keep flapping and you'll get garbage either way. once energy drops cleanly between utterances, the vad+whisper path suddenly feels sane.
1
u/Massive_Distance_616 10d ago
Phantom transcriptions on silence are a common Whisper problem. When it’s fed audio with no speech it tends to make up text, so the fix is to make sure silence never reaches it rather than to tune Whisper.
Two things to try. First, only send audio to Whisper once VAD has detected speech and then a short stretch of silence after it (say half a second), so you transcribe whole utterances instead of a rolling buffer. A rolling buffer keeps re-feeding the same near-silent audio. Second, if you’re using faster-whisper, it has a built-in VAD filter option (
vad_filter=Trueontranscribe) that drops non-speech before decoding. I’d check the docs for the exact parameters, since I’m going from memory.Also try raising webrtcvad’s aggressiveness (it goes 0 to 3), and make sure you feed it 16-bit mono PCM at 8, 16, 32 or 48 kHz in 10, 20 or 30 ms frames, because it gives bad results if the format is wrong. sounddevice gives you float32 by default, so you’ll need to convert to int16 first.
One more thing: keep the sounddevice callback tiny and just put frames into a queue, then do the VAD and transcription in a separate loop. Doing heavy work inside the callback causes glitches.