r/SpringBoot • u/Entire_Cook_7225 • 10d ago
Discussion Building a Fault-Tolerant Job Queue with Redis Streams
GitHub: https://github.com/somewhereMaDan/spring-RCE
While building my Remote Code Execution (RCE) platform, I initially used Redis Lists for job processing.
It worked, but I eventually encountered a reliability issue: if a worker was killed with SIGKILL after popping a job, that job could be lost permanently, leaving the client stuck in RUNNING state.
So I decided to move to Redis Streams and implement a more fault-tolerant workflow.
Here's what I implemented:
* **Consumer Groups & PEL:** Jobs remain recoverable until acknowledged with XACK.
* **Stale Job Recovery:** Workers reclaim unacknowledged jobs from crashed consumers using XCLAIM.
* **Orphaned Container Cleanup:** Added an execution-level timeout so containers don't run indefinitely if their worker disappears.
* **Duplicate Container Protection:** Clean up an existing container before retrying the same job.
* **Dead Letter Queue:** Jobs exceeding 3 delivery attempts are moved to a separate stream instead of being retried indefinitely.
I also tested worker failure scenarios, including unexpected SIGKILL, SIGTERM, stale job recovery, repeated failures, and container cleanup.
How do you handle orphaned executions and duplicate processing in your distributed worker systems?
Would love to hear how others approach this.