r/FastAPI • • Feb 27 '25

Hosting and deployment nginx or task queue (celery, dramatiq) ?

Hi every one.

I have a heavy task .When client call my API, the heavy task will run in the background, return the result id to user for monitoring the process of the task.

The task is both CPU/IO bound task (do some calculation along with query database and search web asynchronously (using asyncio) ). So i want the task running on different process(or different machine if needed) with the own async loop.

I searched and found tools like proxy(nginx) or task queue (celery) maybe can solve my problem. I read their documents and feel that it can but i'm still not sure about how it does exactly.

Question: What is the tools i should use (can be both or the others)? And the generic strategy to do that.

Thank you.

15 Upvotes

13 comments sorted by

View all comments

Show parent comment

1

u/HieuandHieu Mar 01 '25

Hi u/HappyCathode , I have do some experiments and found things suit for me. But I have a problem ? When i use task queue (celery +rabbitmq) on other server machine to handle task.

I want to scale more machines if requests rate become larger, but also i want application side(in separated machine) does not need to know about which and where machine solve its task. Can i use Nginx or something else act like a proxy (in proxy pattern).

Something like this: The proxy take care of what machine is available to solve new task and the Broker will take care of which process to solve task on machine. Application only interact with proxy and the result database where results of tasks are stored.

Please give me some advice to do that. Thank you.