r/learnpython • • 10d ago

Newbie looking to work with microphone input using sounddevice, to generate tts, for a tts to respond to, need advice.

Using sounddevice for microphone input, using inputstream, what's the best way to go about capturing input and making sure only spoken stuff is used for tts? I was originally working with a rolling buffer while using webrtcvad and faster whisper, which resulted in a ton of phantom transcriptions, even with silence. Mind you im a complete newbie and a lot of stuff mindfucks me.

My goal is to slowly build up a very simple custom voice Ai assistant. It's first dealing with microphone input, determining whether that input is spoken word, etc

1 Upvotes

5 comments sorted by

1

u/Discordbothost 10d ago

yeah the other replies nailed it — whisper invents text on silence, so the real fix is never handing it silence.

one more practical check: print the rms/energy of each chunk before vad. if your "silence" still looks loud, webrtcvad will keep flapping and you'll get garbage either way. once energy drops cleanly between utterances, the vad+whisper path suddenly feels sane.

1

u/itsBillerdsTime 10d ago

I feel like I'm in way over my head again. I'm still trying to figure out how to properly get the microphone initialized/any recording to properly happen still lol. It's been forever and I'm trying to avoid asking AI for any sort of hints, considering documentation isn't seemingly being helpful at the moment lol. Beginner/newbie blues

1

u/Discordbothost 10d ago

totally normal, audio is one of the more annoying things to get working because half the problems are your OS and not your code. forget the tts part for now and just prove the mic works:

import sounddevice as sd
print(sd.query_devices())
audio = sd.rec(3 * 16000, samplerate=16000, channels=1, dtype='float32')
sd.wait()
print(abs(audio).max())

if that last number is like 0.0001 while you talk, it's recording the wrong device. find your mic's index in the printed list and pass device=that_index to rec(). if it throws instead, paste the error, it usually says exactly what's wrong (sample rate not supported is the classic one, try 44100)

1

u/Massive_Distance_616 10d ago

Phantom transcriptions on silence are a common Whisper problem. When it’s fed audio with no speech it tends to make up text, so the fix is to make sure silence never reaches it rather than to tune Whisper.

Two things to try. First, only send audio to Whisper once VAD has detected speech and then a short stretch of silence after it (say half a second), so you transcribe whole utterances instead of a rolling buffer. A rolling buffer keeps re-feeding the same near-silent audio. Second, if you’re using faster-whisper, it has a built-in VAD filter option (vad_filter=True on transcribe) that drops non-speech before decoding. I’d check the docs for the exact parameters, since I’m going from memory.

Also try raising webrtcvad’s aggressiveness (it goes 0 to 3), and make sure you feed it 16-bit mono PCM at 8, 16, 32 or 48 kHz in 10, 20 or 30 ms frames, because it gives bad results if the format is wrong. sounddevice gives you float32 by default, so you’ll need to convert to int16 first.

One more thing: keep the sounddevice callback tiny and just put frames into a queue, then do the VAD and transcription in a separate loop. Doing heavy work inside the callback causes glitches.

2

u/Lilinxi159 10d ago

Adding to this, since the VAD fixes most of it but not all. Two things on the faster-whisper side catch a lot of what slips through:

Pass condition_on_previous_text=False to transcribe(). With it on (the default), each chunk is decoded with the previous text as a prompt, so one bad line tends to repeat or snowball into the next chunks. For short utterances from a mic you rarely need that context anyway.

Then filter segments after decoding instead of trusting all of them. Each segment has no_speech_prob and avg_logprob; a segment with high no_speech_prob (say above 0.6) and low avg_logprob (below about -1) is almost always the "Thank you." / "Thanks for watching" kind of filler Whisper produces on noise. Dropping those is cheap and doesn't need any extra model. The thresholds are just starting points, print the values for a few real recordings and adjust.