Pool three idle machines into one LLM endpoint you own

We have machines scattered around: a server box that runs all the time, a Mac for daily work, and now and then a rented pod with a 5090 GPU. The problem is each one runs its own models, so using one means remembering which model sits on which machine, on what port. More hassle than it should be.

Four days ago I came across a repo called mesh-llm that proposes exactly the fix: pull every machine into one pool and expose it as a single place to call. So I forked it and deployed it for real across three machines at home, to see whether it does what it advertises.

Up front: this one is different from our earlier repo reviews because we actually ran it, not just read the code. The internal architecture comes from reading the code and docs; the deployment results and the gotchas come from running it myself across three machines. Three parts: first what mesh-llm is, then what the real deployment surfaced, and finally who it's for and who it isn't.

Part 1What mesh-llm is

If you want to run a model on your own machine, most people reach for Ollama, which is great at running on one box. mesh-llm plays a different game. It pools GPU and memory across several machines and exposes them as one place to call, at localhost:9337/v1, looking exactly like OpenAI's API. Any tool or code already wired for OpenAI just points here, no changes needed.

The charm is that it grows with you. Start with one machine, add another later and pull it in. Every machine exposes the same /v1 surface, so you can call from anywhere. When you send a request, it looks at the model name, decides which machine can serve it, and routes there for you.

What sets it apart

• One machine is enough, then don't spread. If a model is small enough to fit one box, it serves there and skips all cross-machine traffic.
• Too big for one box? Split into stages. The piece called Skippy stage split slices a model into layer stages spread across machines, so several small machines can pool up to run a large one.
• Broad model support. Qwen, Llama, Gemma, Mistral, DeepSeek, GLM and many more families, including vision and OCR models.
• Public or private, your call. A private mesh links machines quietly with an invite token; a public mesh, where you share spare compute with others, announces itself over the Nostr network.

It's written in Rust, genuinely open source under Apache-2.0, and installs with a one-line command. It's by the Mesh-LLM project, and at the time of writing it has around 2,800 stars (checked live on GitHub). All of that is what you get from reading the code and docs. The more interesting part is what happens once you actually run it.

Part 2What the real three-machine deployment surfaced

We linked three machines that could hardly be more different: a server box with only a CPU, a rented 5090 GPU on a pod, and a Mac running on its own chip (metal). We made the box the always-on hub, serving a small Qwen3 0.6B model, and had the Mac join as a client-only node with a single invite token.

The real test: open a terminal on the Mac and fire a curl at localhost:9337/v1. The reply came back, and it was generated on the box, even though we called from the Mac. It works as advertised. Call from any machine and it feels like talking to one unit, without knowing which machine the model actually sits on.

The gotchas we hit (real, from running it)

Getting there hit a few snags worth flagging if you're going to try it.

1. The scary-looking address in the invite token is fine. The invite token carries an internal docker address that other machines can't see. At first I thought it would break, but its transport (iroh) is smarter than that: it starts by talking through a relay, then upgrades to a direct connection (measured around 213 ms). So the internal address baked into the token caused no problem at all.
2. Qwen3's thinking mode eats the token budget until the answer vanishes. Qwen3 likes to "think" before answering, which burns through the set token budget, leaving the actual answer empty. The fix is to turn thinking off (a /no_think mode) or just set a big token budget.
3. The big model costs extra. The box serves the small 0.6B for free all the time, but wanting the 27B means spinning up the rented GPU pod, which costs by the hour, so you shut it down when done. That's a wallet issue, not a limit of the mesh itself.

What the deployment actually changed was the feeling of it. Instead of remembering which model is on which machine and on what port, it became point everything at one place and let the mesh route. And the data never has to leave, because every machine is your own.

Part 3Who it's for, and who it isn't

Honestly: mesh-llm shines only once you have more than one machine. With a single machine, Ollama is simpler and does the job. It's also infrastructure work, so expect to set things up rather than download-and-go. And model support comes at different levels of certification: some are stable, some you'll want to test.

A fit if you're

• Someone with more than one idle machine. A gaming rig with a GPU, an old server box, a spare laptop, pool them all into one.
• Someone who wants one OpenAI endpoint for the whole house. Point every tool at one place, no remembering what lives where, and the data never leaves your machines.
• Someone who wants to run a model too big for one box. Skippy stage split lets several small machines pool up to run it.

Not yet, if you

• Have a single machine. Ollama fits better, no mesh to fuss with.
• Want plug-and-go. This still asks you to set up and understand what you're wiring together.

What to borrow

Even if you never install mesh-llm, the idea worth taking away is this: put one OpenAI-shaped endpoint in front, and let it route to whichever backend can serve, with a simple rule that one machine is enough until it isn't, and you only spread when you have to. That pattern drops onto any backend, it doesn't have to be mesh-llm.

The bigger point beyond the tool: a lot of us are sitting on compute that idles without us noticing. A gaming rig left on, an old laptop in a drawer. A tool like mesh-llm turns that scattered capacity into one LLM endpoint you own, with the data staying home and nothing rented.