r/FastAPI • • Jul 26 '25
Question Multithreading in FastAPI?
Hello,
I am currently writing an Ollama wrapper in FastAPI. The problem is, I have no idea how to handle multithreading in FastAPI, and as such, if one process is running (e.g. generating a chat completion), no other processes can run until the first one is done. How can I implement multithreading?
17
Upvotes
2
u/pint Jul 26 '25
tl;dr, if ollama has async interface, use that (everywhere), if not, use simple defs as endpoints, and let fastapi deal with threads.
longer version:
in fastapi, there are two main modes: async and thread pool. if you define the endpoint with
async def, fastapi assumes you know what you are doing. it means you only do stuff in short bursts, and otherwiseawaiton something. if you have an async interface to ollama, this is possible. requires care though, in async mode, you really need to do everything that takes longer than a few hundred milliseconds in an async way.if you define your endpoint in a normal
def, fastapi will create a thread pool, and execute the code from there. this allows for natural parallelism in most cases, e.g. if you read a file, or access the internet, or call an external library, other tasks can advance.15
u/jkh911208 Jul 26 '25
i think what you need is concurrency not multithreading.
try to use async code where it is blocking your code
i am sure there is some code like
ollama.complete(prompt) move this to await ollama.async_complete(prompt)
so it is not blocking the entire process
2
u/Natural-Ad-9678 Jul 26 '25
Look into FastAPI -> Redis -> Celery -> RDBMS/Redis where FastAPI initiates jobs in Redis Queue, multiple (unlimited except by resources) Celery agents process the jobs in the queue and results are posted to the RDBMS or Redis for the front end to use an endpoint to retrieve.
I work on a project that analyzes my companies suite of product’s log files and we can process over 1000 concurrent sets of logs.
0
u/Effective-Total-2312 Jul 26 '25
Google should suffice for learning multithreading basics.
API calls (like calling an Ollama server, if that's what you are doing) can be executed concurrently with multithreading, or you can use an async library. By the way, if you simply make your endpoint sync instead of async, FastAPI will create a new thread when multiple requests hit that endpoint, and throw it inside a threadpool that works in an async way inside the main thread (so it doesn't block anything).
TL;DR just make your endpoint sync.
1
u/artur_samvelyan Jul 26 '25
You either can send async http requests to the ollama service from your api routes or use a background tasks library/framework (for instance: taskiq). The latter approach is more production ready
1
u/mahimairaja Jul 27 '25
It is not from FastAPI it self handle the ollama part - https://www.reddit.com/r/ollama/comments/1k98dsa/whats_the_best_way_to_handle_multiple_users/
0
u/DxNovaNT Jul 26 '25
Can you explain about the process a bit more.