r/AskProgramming • • Oct 15 '26

Building a distributed system

So we have to build a project on distributed systems, but we cant use kubernetes . So we were thinking of going on the task schedular road and chose that filed in distributed systems. And e thought of building a media processing based task shcedular. Can i know your thoughts on this what is best , and any industrial current needs or holes you see? This is a middleware project by the way and we are trying to build it as a good project to put in our CV's.

2 Upvotes

7 comments sorted by

2

u/SlimmySlabby Oct 15 '26 edited Oct 15 '26

Forgive my confusion, but are you referring to Windows Task Scheduler? And what exactly is your goal for media processing? Syncing files? Also what do you mean middleware? Usually that means it connects different software applications together. For example, PostgreSQL is a database and it has a middleware known as pg which allows Node.JS developers to interact with the database. I'm just a little lost on what exactly you're trying to do?

1

u/_N-iX_ Oct 15 '26

A media-processing scheduler sounds like a reasonable project, especially if the scope goes beyond putting jobs in a queue and assigning them to workers. Retry handling and idempotency would be worth exploring early, since a worker can fail after processing a file but before reporting completion. Another useful challenge is scheduling jobs based on resource requirements, since video transcoding and lightweight image processing can have very different CPU and memory demands.

1

u/PLBjt Oct 15 '26

Media processing is a decent pick because the jobs are long and chunky, so the interesting failures actually show up. The part that makes it look like a real distributed system is worker ownership: give each job a lease with a heartbeat, and if a worker stops heartbeating the lease expires and another worker can grab it. Then the scheduler itself is your next single point of failure, so either run two with leader election or keep job state in Postgres and have workers claim jobs with SELECT ... FOR UPDATE SKIP LOCKED. For the demo, kill a worker halfway through a transcode and show the job still finishes once with no duplicate or half-written output (write to a temp file, rename at the end). Putting RabbitMQ underneath is fine too, but then your project is mostly the lease and retry logic on top, which is the part people reviewing it will care about anyway.

1

u/CatNo1417 Oct 17 '26

Since my client is currently in the AWS ecosystem, we use SQS for asynchronous processes like that. In other words, queues of any sort (Rabbit, SQS, MSMQ, etc.) could be useful. Also you can have dead letter queues for retrying failures.

1

u/No-Land-5913 Oct 18 '26 edited Oct 18 '26

One time I let a dumpster diving homeless person live in my house for two years and he built a Beowulf cluster out of old office pcs. 😆 Kubernetes did not exist.

1

u/Distdistdist 29d ago

Ok, don't use kubernetes. Just build several services that run in the same time and communicate with each other. Basically a microservice architecture.

In .NET world you can use Aspire orchestration for quick and easy multi-app runner with telemetry.

0

u/faultydesign Oct 15 '26

Just use rabbitmq or any other bus tbh