My last post covered how to self-host AI models on a Raspberry Pi 5. This one is about what happens when you try to build actual software with a 3B parameter model running on that Pi. Not a demo. Not a proof of concept. Real software that ships, serves requests, and earns money.
This is also a story about decentralizing AI compute. Not in the abstract, not as a philosophy, but as a practical engineering decision. When you move 70% of your AI workload off cloud APIs and onto an $80 computer in your house, you are decentralizing AI. The question is whether it actually works. After three months, I can tell you what does and what doesn’t.
I run an AI agent on a Raspberry Pi 5. It manages email, writes and deploys code, drafts articles, monitors services, and handles infrastructure. The agent uses a mix of local and cloud models. The local model is a 3B-parameter quantized LLM. The cloud model handles anything requiring deeper reasoning. This split is the most interesting engineering decision I’ve made, and it’s the one I get asked about most.
The Local-Cloud Split
About 70% of my agent’s work runs on the local 3B model. The other 30% goes to a cloud API. The split is not random. It’s based on which task types each model handles well.
Local model (3B, quantized, on the Pi):
- File operations: create, edit, move, delete. 95% success rate.
- Package management: apt install, pip install, npm install. 92% success rate.
- Service management: systemctl start/stop/restart, basic config generation. 88% success rate.
- Email triage: sorting inbox by sender, subject, priority. 85% success rate.
- Web research: fetching pages, extracting specific data points. 80% success rate.
- Code generation under 50 lines: 75% success rate.
Cloud model (large, via API):
- Code generation over 50 lines
- Code review across multiple files
- Article drafting and editing
- Any task requiring reasoning about system architecture
- Anything creative or persuasive
The pattern is clear: the local model handles procedural, deterministic work. The cloud model handles work requiring judgment or creativity. The line between them is the line between “follow instructions” and “make decisions.”
Why Not Just Use the Cloud Model for Everything?
Cost and latency.
The cloud model costs roughly $0.02 per request. My agent makes 50-100 requests per day. That’s $1-2/day, or $30-60/month. Running the local model for 70% of those requests cuts the cloud bill to $10-15/month.
But the bigger reason is latency. A local model on the Pi responds in 2-5 seconds. The cloud API takes 8-15 seconds, sometimes longer during peak hours. When the agent is executing a 10-step deployment checklist, 10 cloud calls at 12 seconds each is two minutes. Ten local calls at 3 seconds each is 30 seconds. For procedural work, the quality difference is negligible, and the speed difference is significant.
There’s also a reliability argument. The Pi is in my house. The cloud API is somewhere else. If my internet drops, the local model keeps working. The agent can still manage local files, run local services, and execute local tasks. It just can’t do research or write articles until connectivity returns. That’s an acceptable degradation.
The Failure Modes That Actually Matter
I’ve seen a lot of discussion about small model “reasoning capability” and “benchmark performance.” In practice, the benchmarks don’t predict agent reliability. What matters is how the model fails, not how well it scores on multiple-choice questions.
Confident wrongness is the big one. The 3B model will state incorrect facts with full confidence. If you ask it whether a package is installed, it might say yes based on the package being in the apt cache, even if it was never actually installed. The solution is never to ask the model about system state. Always check the system state directly and give the model the result as input.
Instruction drift is the second. In a multi-step task, the model gradually shifts its interpretation of the original instruction. Step 1 is exactly what you asked. Step 3 is close. Step 5 is adjacent but not the same. Step 8 is doing something related but different. The solution is to restate the goal at each step, not trust the model to hold it in context.
Format inconsistency is the third. The model outputs JSON 90% of the time, but 10% of the time it wraps it in markdown code blocks or adds commentary before or after. If your parser expects strict JSON, this breaks. The solution is a tolerant parser that strips markdown fences and extracts the JSON block, rather than expecting the model to be perfect.
None of these failure modes show up in benchmarks. All of them show up within the first hour of running an agent.
The Architecture That Works
After three months of iteration, here’s the architecture I’ve converged on:
User request
|
v
Task planner (cloud model)
-- breaks request into atomic steps
-- each step has: action, expected result, verification
|
v
Step executor (local model)
-- runs one step
-- returns result + output
|
v
Verification (deterministic, no model)
-- checks system state against expected result
-- if pass: next step
-- if fail: report to planner
|
v
[repeat until all steps complete or failure]
The cloud model plans. The local model executes. Deterministic code verifies. No model evaluates its own output.
This architecture costs more API calls than a pure-local approach (the planning step goes to the cloud), but it’s dramatically more reliable. The local model never has to hold a complex plan in context. It just executes one step at a time. The cloud model never has to execute commands, which it’s bad at anyway. And the verification layer catches errors that both models miss.
What I Built With This
I’m not going to list every project, but here are the categories of real software this setup has produced:
- A domain availability checker with an API back end, running on the Pi behind a tunnel, querying RDAP endpoints across 500+ TLDs. The local model handles the deployment, monitoring, and log rotation. The cloud model wrote the initial codebase.
- Article pipeline: the agent drafts articles (cloud model), formats them, and submits to publications. The local model handles the submission mechanics, email notifications, and scheduling.
- Infrastructure monitoring: the local model checks service health every few minutes, restarts failed processes, and sends alerts. No cloud involvement at all. This runs entirely on the Pi.
- A media player customisation tool with custom on-screen controls. The cloud model handles the UI design and user-facing code. The local model handles build scripts, testing, and release packaging.
None of these are toy projects. The domain checker processes real queries. The article pipeline has published 80+ articles. The monitoring has caught and recovered from real outages. The media player has real users.
The Cost Breakdown
Here’s what this setup actually costs per month:
- Raspberry Pi 5 (8GB): $80 one-time, amortised to near zero
- NVMe SSD (512GB): $35 one-time (admittedly now more expensive)
- Cloud API costs: $10-15/month
- Electricity: ~$2/month (the Pi draws 5-8 watts)
- Internet: already had it
- Total ongoing: $12-17/month
For comparison, running everything on a cloud API would cost $40-60/month. Running everything on a cloud GPU instance would cost $100+/month. The Pi pays for itself in 2-3 months.
Lessons Learned
If I were starting over, three things:
- Start with the verification layer, not the model. I spent weeks tuning prompts and model parameters before realising that deterministic verification was more important than any prompt engineering. If I’d built the verification first, I would have caught failures sooner and wasted less time debugging model behaviour.
- Use the cloud model for planning from day one. I tried to do everything locally at first. The 3B model is not good at breaking complex requests into steps. It misses dependencies, orders steps incorrectly, and forgets constraints. Offloading just the planning step to the cloud model improved reliability by 30%+.
- Truncate tool outputs more aggressively. I was sending full command output back to the model. 500-line log files, full directory listings, complete API responses. The model made worse decisions with more data, not better. Trimming to the last 20 lines plus a one-line summary improved success rates on multi-step tasks by 15%.
Assessment Summary
A 3B model on a Raspberry Pi is not a replacement for a cloud API. It’s a complement. If you’re building an agent and trying to choose between local and cloud, the answer is both. Use the local model for the 70% of work that’s procedural. Use the cloud for the 30% that requires reasoning. The engineering challenge is drawing that line correctly, and the only way to find it is to try.
The benchmark gap between a 3B model and a 70B model is real. The practical gap is smaller than you think, if you architect around it. Most agent work is not reasoning. Most agent work is execution. A 3B model can execute. It just needs the right structure around it.
Decentralized AI isn’t just about running nodes on a blockchain or training models across federated clusters. It’s about who controls the compute. When 70% of my agent’s work runs on hardware I own, in my house, without depending on a cloud provider’s pricing or uptime, that’s decentralization in the most literal sense. The Pi 5 is not a server rack. It’s not a GPU cluster. It’s a $80 credit-card-sized computer that handles a surprising amount of real work. The question was never whether small models are good enough. The question is whether we’re willing to do the engineering work to make them useful. Three months in, my answer is yes.