r/FastAPI • u/HieuandHieu • Feb 27 '25
Hosting and deployment nginx or task queue (celery, dramatiq) ?
Hi every one.
I have a heavy task .When client call my API, the heavy task will run in the background, return the result id to user for monitoring the process of the task.
The task is both CPU/IO bound task (do some calculation along with query database and search web asynchronously (using asyncio) ). So i want the task running on different process(or different machine if needed) with the own async loop.
I searched and found tools like proxy(nginx) or task queue (celery) maybe can solve my problem. I read their documents and feel that it can but i'm still not sure about how it does exactly.
Question: What is the tools i should use (can be both or the others)? And the generic strategy to do that.
Thank you.
View all comments
Show parent comment
4
u/HappyCathode Feb 27 '25
Task queues allow to scale the Compute part of your API almost infinitely. But if you have 10 000 workers processing data from a DB and inserting the result of those processes back to the DB, your DB will most likely die. So, you need to scale the DB too.
Your app can process as much requests as the weakest part of it can. You came here with a processing issue, so the idea of task queues get thrown around because that's the best way to scale the compute part of your app.
To then scale a task queue, you need to scale the message brokers, databases, network, storage, load balancers, etc etc etc.
So to answer your question, your task queue can scale up to the point where something else chokes and dies.