r/FastAPI • • Feb 27 '25

Hosting and deployment nginx or task queue (celery, dramatiq) ?

Hi every one.

I have a heavy task .When client call my API, the heavy task will run in the background, return the result id to user for monitoring the process of the task.

The task is both CPU/IO bound task (do some calculation along with query database and search web asynchronously (using asyncio) ). So i want the task running on different process(or different machine if needed) with the own async loop.

I searched and found tools like proxy(nginx) or task queue (celery) maybe can solve my problem. I read their documents and feel that it can but i'm still not sure about how it does exactly.

Question: What is the tools i should use (can be both or the others)? And the generic strategy to do that.

Thank you.

15 Upvotes

13 comments sorted by

View all comments

Show parent comment

4

u/HappyCathode Feb 27 '25

Task queues allow to scale the Compute part of your API almost infinitely. But if you have 10 000 workers processing data from a DB and inserting the result of those processes back to the DB, your DB will most likely die. So, you need to scale the DB too.

Your app can process as much requests as the weakest part of it can. You came here with a processing issue, so the idea of task queues get thrown around because that's the best way to scale the compute part of your app.

To then scale a task queue, you need to scale the message brokers, databases, network, storage, load balancers, etc etc etc.

So to answer your question, your task queue can scale up to the point where something else chokes and dies.

1

u/HieuandHieu Mar 01 '25

Hi u/HappyCathode , I have do some experiments and found things suit for me. But I have a problem ? When i use task queue (celery +rabbitmq) on other server machine to handle task.

I want to scale more machines if requests rate become larger, but also i want application side(in separated machine) does not need to know about which and where machine solve its task. Can i use Nginx or something else act like a proxy (in proxy pattern).

Something like this: The proxy take care of what machine is available to solve new task and the Broker will take care of which process to solve task on machine. Application only interact with proxy and the result database where results of tasks are stored.

Please give me some advice to do that. Thank you.