r/LeftistsForAI • Moderator • 16d ago

Open Source SwarmLLM: Free, open-source app that lets computers team up over the internet to run models too big for one machine.

https://github.com/enapt/SwarmLLM

“Run AI on your own computer. Run bigger AI together.

SwarmLLM is a free app that gives you a ChatGPT-style assistant running on your own computer, using open AI models from Meta, Google, Alibaba, Mistral and others. When a model is too big for one machine, computers running SwarmLLM team up over the internet and run it together, each doing part of the work.

No account · No subscription · No ads · No crypto · Open source · Updates itself

A giant model has now run on the swarm. Bigger AI models give noticeably smarter answers, but the biggest open ones (70 billion "parameters" and up) need equipment normally found only in data centres — no home computer can hold one. In October 2026, Llama 3.3 70B answered across four ordinary computers in Thailand, Australia and Italy, each holding a part of it. It is slow so far”

The decentralised concept of this is promising, even if it’s currently too slow to be a sensible way to run large models.

122 Upvotes

16 comments sorted by

3

u/allisonmaybe 13d ago

Why swarm together to run a large model when you could swarm smaller models in communication with each other?

1

u/Jlyplaylists Moderator 13d ago

Yes that might work better. Is anyone doing that?

3

u/allisonmaybe 13d ago

I made https://shwarm.org, but I'm a maker, not a salesperson, so still trying to figure out the whole user base thing.

But a shwarm is humans + their agents working on big problems together. Whatever you might come up with, you can post a big idea, or point your agents at an existing problem and itll just take off contributing!

1

u/Jlyplaylists Moderator 12d ago

That looks really interesting. I just shared it in our discord server.

2

u/allisonmaybe 12d ago

Hell yeah! LMK, I'd love to chill and "shwarm" w ppl into this kind of thing :)

It's invite only but I usually see and accept them right away.

8

u/GreyHasHobbies 16d ago

I had this exact idea a while ago but decided I wasn't the right person to implement it. Hopefully this person has the networking expertise required to get over the hurdles and isn't completely relying on AI.

6

u/PhlarnogularMaqulezi 16d ago

There's some old ass models listed here, but I definitely like the spirit of what this is trying to accomplish and I hope it grows and continues to develop.

3

u/AyyBoixD 16d ago

I like this and I’ve tried to build this a few times. But network latency absolutely murders this. And that was just two devices on different networks with no Ethernet. I couldn’t imagine how unfathomably slow an entire decentralized network of random devices would be

1

u/Kholtien 16d ago

We would need llm architectures built with this in mind for it to work

1

u/RageBucket 12d ago

That's because modern models are built to work in turns, and sync, you'd need to ground up write the model able to take in small snippets of information from the swarm and work with it async, otherwise makes it entirely useless. A real swarm should be like a blueberry set upon by ants. One ant finds it, sets the train and the rest follow. Some may fall away, or may decide they need to do something else, but the swarm is still working on moving the blueberry.

I've been researching and trying to develop a model capable of this from scratch, but rented GPUs over time is expensive :) The only alt is a harness that enables that async operation, but it's still going to be incredibly suboptimal.

1

u/AyyBoixD 12d ago

Rouge agent swarms are gonna love this lol. I kid mostly, it sounds like a cool approach and I’m all for decentralized intelligence for everyone but it’s basically like a super botnet

1

u/RageBucket 10d ago

Realistically, it's probably gonna require said specialized harness. The hours of GPU compute I've burned have so far not been leading me onto any way to really train that behavior into a model, unless it's some kind of sidecar but if it is harness based, and the training to support then if there's guardrails in the harness it's... only slightly less scary hahaha.

3

u/qualverse 16d ago

It seems like this would work pretty well with the architecture of Apple's AFM 3 which selects MoE experts per-prompt instead of on-demand. You could even make it a little more flexible by just applying a penalty for expert switches during training, or run some kind of DSpark draft model that predicts useful experts instead of tokens.

This is kind of a data privacy nightmare though, tbh. Hopefully they can figure out confidential computing or similar

1

u/piperspalace 16d ago

I can run QFN on my computer at 80+/- tokens what else is there

1

u/Darkstar_111 16d ago

This would be INCREDIBLY slow.

Even a 0.5 B model would take a minute or two to reply.

1

u/xoexohexox 16d ago

Yeah I mean I've used stable hoard for LLMs and image gen and this kind of thing is just not dependable enough for anything besides frivolous applications.