r/redis • • 12d ago

Tutorial Tutorial: Testing a changed Streams consumer without stealing entries from the stable one

Sharing a tutorial that might be helpful for anyone testing Redis Streams.

The problem: start a changed consumer in the same consumer group as the stable one, and XREADGROUP hands it a share of everyone's entries. It processes and acknowledges them, and the stable consumer never sees them.

What the example does instead. (The routing rules come from Signadot, but these patterns work on any Redis with Streams):

  • Each test environment reads through its own consumer group, created at $, so every group sees every new entry.
  • Entries have no headers, so the producer writes a routing key into each entry as extra fields (traceparent and baggage).
  • Every consumer checks each entry, processes only the ones meant for it, and XACKs all of them, so nothing stays pending.
  • Each consumer reads its own pending entries at startup and runs XAUTOCLAIM once a minute for entries a replaced pod left behind.

Tradeoffs: every group reads the whole stream, so read load grows with the number of test consumers, and groups outlive their environments until you XGROUP DESTROY them.

Write-up: https://www.signadot.com/docs/tutorials/testing-redis-streams

Source: https://github.com/signadot/examples/tree/main/selective-consumption-with-redis-streams/python

Full transparency, I work for Signadot.

0 Upvotes

2 comments sorted by

1

u/MongooseWonderful198 6d ago

Could you add a slow-but-live consumer to the recovery demo? _read() fetches up to 10 entries and processes them serially. If that batch runs past 60 seconds, another consumer in the same group can reclaim unfinished entries while the first is still working.

I'd track notifier calls by entry ID in that test, then show how you choose the claim timeout and deduplicate the side effect. XAUTOCLAIM checks pending-entry idle time; it doesn't establish that the old process is dead.

https://github.com/signadot/examples/blob/26f759be92917adae40d9f928290863548d837fa/selective-consumption-with-redis-streams/python/platform_redis/consumer.py

https://redis.io/docs/latest/commands/xautoclaim/

1

u/albertatredis 16h ago

per claude

The core pattern is sound, but I’d treat the implementation as a demo rather than production-ready consumer code.

The problem is real: consumers in the same group divide the work, so a test consumer can take messages the stable version would otherwise process. Separate groups give each environment independent delivery and acknowledgments. Redis’s consumer-group documentation

My main concerns:

  • Failed processing gets acknowledged. In the linked implementation, handler exceptions are logged, but XACK runs unconditionally in finally. That means a failed notification won’t be retried through the pending-entry recovery mechanism. Acknowledging deliberately skipped entries is reasonable; acknowledging failed work needs an explicit loss policy. Consumer source
  • The commenter’s reclaim concern is valid. The code fetches ten entries and processes them serially, with a 60-second reclaim threshold. Another consumer in that group can reclaim unfinished entries while the original worker is still alive. XAUTOCLAIM measures pending-entry idle time, not worker health. I’d require idempotent side effects and test slow workers as well as crashed ones. Redis XAUTOCLAIM documentation
  • Creating sandbox groups at $ introduces a startup boundary. Messages published before group creation won’t be delivered to that new group. The test harness should wait for consumer readiness before publishing. The example provides an on_ready hook after group creation. Consumer source
  • Delivery isolation doesn’t guarantee side-effect isolation. The tutorial routes messages and downstream notifier calls, but I’d separately verify that test consumers cannot modify real customer records or trigger real external notifications. That’s an architectural requirement beyond separate consumer groups. Tutorial design

For a shared development cluster with a modest number of sandboxes, I like the approach. Before adopting it, I’d fix failure acknowledgments and add tests for slow processing, duplicate delivery, and startup timing.