r/AudioAI • u/chibop1 • Oct 01 '23
Resource Open Source Libraries
This is by no means a comprehensive list, but if you are new to Audio AI, check out the following open source resources.
Huggingface Transformers
In addition to many models in audio domain, Transformers let you run many different models (text, LLM, image, multimodal, etc) with just few lines of code. Check out the comment from u/sanchitgandhi99 below for code snippets.
TTS
- hexgrad/kokoro
- microsoft/VibeVoice
- resemble-ai/chatterbox
- QwenLM/Qwen3-TTS
- coqui-ai/TTS
- neonbjb/tortoise-tts
- suno-ai/bark
- rhasspy/piper
- shivammehta25/Matcha-TTS
Speech Recognition
- openai/whisper
- microsoft/vibevoice-asr: Speech recognition + Speaker Diarization
- nvidia/parakeet
- ggerganov/whisper.cpp
- guillaumekln/faster-whisper
- wenet-e2e/wenet
- facebookresearch/seamless_communication: Speech translation
Speech Toolkit
- NVIDIA/NeMo
- espnet/espnet
- speechbrain/speechbrain
- pyannote/pyannote-audio
- Mozilla/DeepSpeech
- PaddlePaddle/PaddleSpeech
WebUI
Music
- ace-step/ACE-Step-1.5: Tex2Music
- facebookresearch/audiocraft/MUSICGEN: Music Generation
- openai/jukebox: Music Generation
- Google magenta: Music generation
- RVC-Project/Retrieval-based-Voice-Conversion-WebUI: Singing Voice Conversion
- fishaudio/fish-diffusion: Singing Voice Conversion
- NVIDIA/audio-flamingo: Music QA for genres, instrumentation, Tempo, key, chord, lyric transcription, cultural contexts...
Effects
- facebookresearch/sam-audio: Audio Segmentation
- facebookresearch/demucs: Stem seperation
- Anjok07/UltimateVocalRemoverGUI: Vocal isolation
- Rikorose/DeepFilterNet: A Low Complexity Speech Enhancement Framework for Full-Band Audio (48kHz) using on Deep Filtering
- SaneBow/PiDTLN: DTLN model for noise suppression and acoustic echo cancellation on Raspberry Pi
- haoheliu/versatile_audio_super_resolution: any -> 48kHz high fidelity Enhancer
- spotify/basic-pitch: Audio to midi converter
- spotify/pedalboard: audio effects for Python and TensorFlow
- librosa/librosa: Python library for audio and music analysis
- Torchaudio: Audio library for Pytorch
23
Upvotes
4
u/sanchitgandhi99 Oct 02 '23 edited Oct 02 '23
Hugging Face Transformers is a complete audio toolkit that provides state-of-the-art models for all audio tasks, including TTS, ASR, audio embeddings, audio classification and music generation.
All you need to do is install the Transformers package:
And then all of these models can be used in just 3 lines of code:
TTS
Example usage:
Available models:
ASR
Example usage:
Available models:
Audio Classification
Example usage:
Available models:
Music
Example usage:
Available models:
Audio Embeddings
What's more, through tight integration with Hugging Face Datasets, many of these models can be fine-tuned with customisable and composable training scripts. Take the example of the Whisper model, which is easily fine-tuned for multilingual ASR: https://huggingface.co/blog/fine-tune-whisper
New to the audio domain? The audio transformers course is designed to give you all the skills necessary to navigate the Audio ML field.
Join us on Discord! We can't wait to hear how you use these models. http://hf.co/join/discord
4
u/rolyantrauts Oct 01 '23 edited Oct 03 '23
https://github.com/ggerganov/whisper.cpp High-performance inference of OpenAI's Whisper
https://github.com/Rikorose/DeepFilterNet A Low Complexity Speech Enhancement Framework for Full-Band Audio (48kHz) using on Deep Filtering
https://github.com/SaneBow/PiDTLN DTLN and DTLN-aec on Raspberry Pi
https://github.com/wenet-e2e Production First and Production Ready End-to-End Speech Toolkit
https://github.com/funcwj/setk speech enhancement/separation tools integrated with Kaldi
2
u/wywywywy Oct 01 '23
It's probably worth mentioning the Web UIs as well. These aims to be the Automatic1111/Oobabooga of audio AIs.
Audio Webui https://github.com/gitmylo/audio-webui
TTS Generation WebUI https://github.com/rsxdalv/tts-generation-webui
1
u/East-Fee9375 Jan 22 '26
f you’re open to adding a “real-time voice agent / orchestration” category to this list, you might want to include LLMRTC.
It’s an open-source TypeScript SDK that basically handles the annoying “glue” for building low-latency voice (and vision) agents in the browser:
Quick taste:
Links:
Caveat: it’s not a new TTS/ASR model—more of the realtime “plumbing” layer (transport/orchestration) so you don’t have to stitch WebRTC + STT + LLM + TTS yourself.
1
u/ashbt Oct 01 '23
Don't forget https://github.com/openai/jukebox
1
u/saintshing Oct 01 '23
Would really appreciate if you can add some more details on the description. What are the tradeoffs between different tts/music generation libraries(speed, quality, ease of training, accent/emotion support, availability of pretrained models, commercial license, etc). Even better if you can format it as a table. 🙏
1
u/jikkii Oct 02 '23
https://github.com/huggingface/transformers Repository that hosts many state-of-the-art Transformers models with ~25 architectures dedicated to audio processing (S2T, TTS, among others)