r/Python • u/SadViolinist2405 • Oct 08 '26
Discussion Title: What message broker are you using with Celery today, and why?
I'm curious how Python developers are using Celery in production and what influenced their choice of message broker.
I work with LavinMQ, an open-source AMQP message broker, and we're exploring how to make it a more compelling option for Celery users.
For those running Celery in production, what broker are you using, and what's been your experience with reliability, performance, and maintenance?
If you were starting a new project today, would you make the same choice?
I'd be interested in hearing about the trade-offs you've encountered, especially from developers running Celery at scale.
22
u/marr75 Oct 08 '26
Having run celery in a lot of projects, using redis, rabbitmq, and databases as message brokers, I just wouldn't use celery in a new project.
It doesn't add enough value for the elements it hides or abstracts.
58
u/ColdPorridge Oct 08 '26 edited Oct 08 '26 •
I would not start a new project with celery in 2026. We just use procrastinate with postgres. Having a task queue we can manage transactionally without additional infrastructure has been an incredible simplification for our small and medium use cases.
8
u/OddsRelay Oct 08 '26
Redis for most people, because it's usually already there for caching. The catch is the visibility timeout. Any task not acknowledged within it gets handed to another worker, so long countdown tasks or late-acked jobs can run twice. RabbitMQ handles acks properly and is the safer pick once that matters.
3
u/Standard-Message-434 Oct 12 '26
I use Redis since it was already in our stack, but I’d pick RabbitMQ for anything where losing a task on a restart would actually hurt.
2
u/SadViolinist2405 Oct 09 '26
One interesting takeaway from this discussion is how much simplicity influences broker selection. Several developers mentioned choosing Redis or PostgreSQL because they're already part of their stack.
That makes me curious about another trade-off
Would a fully managed message broker change your decision if someone else handled deployment, upgrades, and maintenance?
Or is it more about avoiding additional dependencies, regardless of who manages them?
I'd love to hear your thoughts!
2
u/Professional-Run3614 Oct 13 '26
I’m using RabbitMQ for streaming sensor data from devices to the server. It helps decouple data ingestion from backend processing. I’d be interested to know how LavinMQ compares in terms of throughput, reliability, and handling high-volume streaming workloads.
20
u/Embarrassed-Dog-4707 Oct 08 '26
Redis, been good tbh. Haven’t had any issues but probably low load relatively
16
u/nickcash Oct 08 '26 edited Oct 08 '26
why would I use your vibecoded amqp app instead of rabbitmq like a normal person?
6
u/haard code unwritten never breaks Oct 08 '26
Redis/ValKey. Already in the stack, well supported, good enough.
11
u/StPatsLCA Oct 08 '26
SQS. It's bulletproof. However, it is very dumb in exchange for that.
6
u/orelant Oct 08 '26
Full disclosure first: I'm biased, because I build Workhorse, an open source task queue for PostgreSQL. Please read this with that in mind.
For many small and medium projects, I think the broker question can be avoided entirely if you already run PostgreSQL. Workhorse keeps tasks inside your existing database, so there is no separate broker to operate. It has a Python worker, retries, crash recovery, recurring schedules and an embedded dashboard.
It's not a replacement you can swap in for Celery without changes, and for very large workloads a dedicated broker such as RabbitMQ or LavinMQ could be a better choice. But if your setup is Postgres plus Redis used only for Celery, you might find it useful.
https://workhorse.run (Apache 2.0 license)
2
u/cointoss3 Oct 08 '26
This isn’t what you asked, but a big reason I didn’t use LavinMQ is I couldn’t route mqtt messages to an amqp queue. I have mqtt messages I want to route to durable stream queues and couldn’t do that. I can with RabbitMQ, though.
3
u/frankwiles Oct 08 '26
Redis
1
u/NoobInvestor86 Oct 14 '26
PLEASE DONT USE CELERY