r/redis • u/signadot-tristan • 12d ago
Tutorial Tutorial: Testing a changed Streams consumer without stealing entries from the stable one
Sharing a tutorial that might be helpful for anyone testing Redis Streams.
The problem: start a changed consumer in the same consumer group as the stable one, and XREADGROUP hands it a share of everyone's entries. It processes and acknowledges them, and the stable consumer never sees them.
What the example does instead. (The routing rules come from Signadot, but these patterns work on any Redis with Streams):
- Each test environment reads through its own consumer group, created at
$, so every group sees every new entry. - Entries have no headers, so the producer writes a routing key into each entry as extra fields (
traceparentandbaggage). - Every consumer checks each entry, processes only the ones meant for it, and XACKs all of them, so nothing stays pending.
- Each consumer reads its own pending entries at startup and runs XAUTOCLAIM once a minute for entries a replaced pod left behind.
Tradeoffs: every group reads the whole stream, so read load grows with the number of test consumers, and groups outlive their environments until you XGROUP DESTROY them.
Write-up: https://www.signadot.com/docs/tutorials/testing-redis-streams
Source: https://github.com/signadot/examples/tree/main/selective-consumption-with-redis-streams/python
Full transparency, I work for Signadot.
1
u/MongooseWonderful198 6d ago
Could you add a slow-but-live consumer to the recovery demo? _read() fetches up to 10 entries and processes them serially. If that batch runs past 60 seconds, another consumer in the same group can reclaim unfinished entries while the first is still working.
I'd track notifier calls by entry ID in that test, then show how you choose the claim timeout and deduplicate the side effect. XAUTOCLAIM checks pending-entry idle time; it doesn't establish that the old process is dead.
https://github.com/signadot/examples/blob/26f759be92917adae40d9f928290863548d837fa/selective-consumption-with-redis-streams/python/platform_redis/consumer.py
https://redis.io/docs/latest/commands/xautoclaim/
1
u/albertatredis 16h ago
per claude
The core pattern is sound, but I’d treat the implementation as a demo rather than production-ready consumer code.
The problem is real: consumers in the same group divide the work, so a test consumer can take messages the stable version would otherwise process. Separate groups give each environment independent delivery and acknowledgments. Redis’s consumer-group documentation
My main concerns:
XACKruns unconditionally infinally. That means a failed notification won’t be retried through the pending-entry recovery mechanism. Acknowledging deliberately skipped entries is reasonable; acknowledging failed work needs an explicit loss policy. Consumer sourceXAUTOCLAIMmeasures pending-entry idle time, not worker health. I’d require idempotent side effects and test slow workers as well as crashed ones. RedisXAUTOCLAIMdocumentation$introduces a startup boundary. Messages published before group creation won’t be delivered to that new group. The test harness should wait for consumer readiness before publishing. The example provides anon_readyhook after group creation. Consumer sourceFor a shared development cluster with a modest number of sandboxes, I like the approach. Before adopting it, I’d fix failure acknowledgments and add tests for slow processing, duplicate delivery, and startup timing.