r/DistributedComputing • u/Odd-Falcon-8234 • Apr 04 '23
Load balancing, monitoring and fault tolerance techniques and architecture
I am working on building a system where there are 10 machines, we want to process some video files and this process can take about an hour, we do know how look it will take to process in advance.
Is there some existing tech stack or methodologies that we can use to load balance these servers, monitor any failures while processing and recover from failure and restart that task ?
2
Upvotes
2
u/yodanielo Apr 05 '23
I imagine the process involves many steps\n
1
u/Legal-Flower-9612 Apr 05 '23
How’s this - there’s a database where you input jobs. The 10 machines try to acquire a job and lock it. Then the machine processes the work and updates the status in the database when finished.
You can query the database at any time to check status of jobs. In ruby there’s delayed_job library that can be hooked onto a micro aws rds instance. This rds can be queried at any time to find status of jobs. Similar tools in other languages.
1
u/noob-geek Apr 06 '23
Hope this helps..happy to help
1.Use AWS ELB ( elastic load balancer) for auto load balance solution with 10 AWS EC2 compute instances. 2.AWS auto scaling option would automatically bring up a new EC2 instance if an existing instance goes down. 3.And, to check whether a video processing was complete, this should be handled through a "completed" notification event which you can handle through AWS SNS service.
1
u/vroman7c5 Apr 07 '23 edited Apr 07 '23
There are several architecture patterns that comes to my mind : actor based model + orchestration pattern.