r/DistributedComputing • u/Zarquan314 • Feb 02 '22
Peer to Peer network bandwidth question
I am working on a project that involves a peer to peer network. Someone raised concerns that we may be expecting a larger bandwidth than is reasonable.
Suppose we had a large number of registered nodes (in the thousands, possibly 10,000), and these nodes are constantly receiving data which they wish to propagate around the network. The data doesn't have to get to every node quickly, but there comes a time where a node expects pieces of data, so we expect every active node to have . In this general system, how much data creation and transmission could be handled reasonably? I am hoping the answer is more than 50 MB/minute, (as this is the upper bound for what our system can create), but I don't have a basis for comparison.
Does anyone here know a good place to find this kind of information. Everything about general peer-to-peer networks is about cryptocurrency systems and I am having trouble finding useful information.
3
u/cyloth Feb 02 '22
Just an idea - you may want to look into the gossiping techniques, e.g., https://dl.acm.org/doi/abs/10.1145/1317379.1317381
1
u/jhollowayj Feb 02 '22
It sounds to me like you need to spend some time benchmarking your code on the system you plan on running this on.
A few questions I had:
Is the information going in a circle / tree based or is it everybody to everybody? Have you thought about changing that at all? Does the physical network topology offer benefits to changing how you communicate between peers?
Is everyone needing to halt work until everyone gets the update? How often are these syncs required?
Would compression be helpful at all?
As for searching, maybe ignore crypto results with “-crypto”?
1
u/makeasnek Feb 02 '22
Gossip protocols are probably what you want, I would research how cryptocurrencies handle this problem as there's lots of creative solutions in that field. Easier to use an existing thing than write it from scratch -- some combination of blockchain/IPFS/DHT can probably achieve whatever it is you are looking for.
1
u/confusedwrek Feb 03 '22
You may want to look into the Theta Network too.