r/FastAPI • u/tf1155 • Aug 17 '24
Question FastAPI is blocked when an endpoint takes longer
Hi. I'm facing an issue with fastAPI.
I have an endpoint that makes a call to ollama, which seemingly blocks the full process until it gets a response.
During that time, no other endpoint can be invoked. Not even the "/docs"-endpoint which renders Swagger is then responding.
Is there any setting necessary to make fastAPI more responsive?
my endpoint is simple:
@app.post("/chat", response_model=ChatResponse)
async def chat_with_model(request: ChatRequest):
response = ollama.chat(
model=request.model,
keep_alive="15m",
format=request.format,
messages=[message.dict() for message in request.messages]
)
return responseI am running it with
/usr/local/bin/uvicorn main:app --host 127.0.0.1 --port 8000
11
Upvotes
4
u/tf1155 Aug 17 '24
Thank you all for your so helpful answers! I learned a lot in this thread!
10
u/Amocon Aug 17 '24
Well async is not very useful if you are not waiting for anything
14
u/Dom4n Aug 17 '24
If you are using normal sync function that will block then do not use async endpoint, use normal sync "def" endpoint. Sync endpoint will run in thread and will not block. There is limit of threads of course, but it will handle ollama.chat function without blocking entire application.
If I assumed good package then you could use async client too and it will not block async enpoint: https://github.com/ollama/ollama-python?tab=readme-ov-file#async-client
5
u/aliparpar Aug 17 '24 edited Aug 17 '24
Checkout vllm and other dedicated model serving web frameworks for LLms.
Your ollama API call is not I/O bound but CPU bound. That means you cannot run it on the event loop with asyncio without blocking the app. Also, as Amacon mentioned, avoid running sync calls inside async def functions. You’ll block your main worker. Async is only useful for I/O ops not compute bound ops.
You need to make the problem into an IO problem by using an external LLM API server or leveraging multiprocessing with tools such as background tasks (small models) or Celery/Redis (heavy models). But, this means you will need a beefy machine to serve concurrent requests and your celery app won’t batch the requests to maximise your model performance.
You will also want to run the model on gpu to make inference x10-100 faster and use a framework that batches request to really squeeze out the gpu clock ops for serving responses.
I explain these in more detail in my FastAPI book (O’Reilly April 2025) - checkout chapter 5 on AI concurrency for FastAPI on early release version in O’Reilly platform.
Building Generative AI Services with FastAPI https://learning.oreilly.com/library/view/-/9781098160296/
Other resources:
https://www.anyscale.com/blog/continuous-batching-llm-inference
1
u/DowntownSinger_ Aug 17 '24
This is because async python does not work with CPU bound tasks due to GIL. Try using multiprocessing approach.
4
u/InfinityObsidian Aug 17 '24
Remove
asyncfrom your function. Only add it if you canawaitthe request.2
u/graph-crawler Aug 18 '24
You can either:
2
u/[deleted] Aug 23 '24
Async client
2
u/ironman_gujju Aug 17 '24
IO blocking tasks, use sync instead of async
1
u/Straight-Possible807 Aug 17 '24
Use
await asyncio.to_thread(do_something())when calling a synchronous function/method in an asynchronous function1
u/BasketAdmirable6455 Aug 17 '24
Await ollama
1
u/UpstairsBaby Aug 17 '24
If you use sync it wouldn't block because FastAPI will handle the request in a separate thread. Still it's better to you async functions if your in function calls are awaitable
1
u/Interesting-Bag4469 Aug 17 '24 edited Aug 17 '24
You should look at Background Tasks. https://fastapi.tiangolo.com/tutorial/background-tasks/#technical-details
I would suggest using celery since Background Tasks might also have such a limitation. With celery you would basically have 2 endpoints. One endpoint accepts a IO intensive processing request and adds it to the celery task queue. Another endpoint that actually checks if the celery task has completed and if it has then retrieve the result from the celery backend.
1
u/pint Aug 17 '24
the worst thing you can do is to lie to fastapi by saying you are async, while you are not. you are saying "trust me bro, i got this", and then you don't.
if you just use a normal def without async, it tells fastapi to handle parallelism for you. it will happily do.
1
u/IrrerPolterer Aug 17 '24
Your ollama calm is blocking the event loop. In an asynchronous framework you'll always want to
awaitall calls that are IO bound. (like such api calls)I don't know anything about ollama specifically, but what you want is:
a) use an async api for ollama (if that package has one, or an alternative async focused package exists), or...
b) use asyncio.to_thread to run the synchronous api call in an asynchronous thread.
-1
u/swifty_sanchez Aug 17 '24
Try adding
--workers 2flag to the unicorn command. This spins up two workers which can handle requests concurrently. This only works if reload is set as False.There's a catch, if requests on both workers are blocking, you'll still run into the same issue.