Local LLMs Got Good. Then AI Outgrew Your Desk.

01 / the scene

The gold box and the feeling of owning the future

Picture the keynote. The lights, the leather jacket, the small machine held up like a relic pulled back from the future. It runs a full AI stack, the man says, the same one that lives in those city-block data centers, only now it sits on your desk and sips less power than a space heater. The room believes him. And honestly, why wouldn't they? The dream is a good one.

So a wave of people went shopping. They bought Mac Studios stuffed with unified memory. They bought AMD Strix Halo mini PCs. They put themselves on the waitlist for an NVIDIA DGX Spark, the gold one, and started dreaming of a private ChatGPT humming quietly under the desk, answering only to them. We get it. There is something deeply satisfying about owning the thing instead of renting it, about a model that is yours: offline, uncensored, with no meter running.

We are not here to dunk on the hardware. These are gorgeous machines, and the engineering inside them is real. We have written before about how NVIDIA quietly stopped being a graphics-card company and reinvented itself as the arms dealer of the AI era, and the Spark is a small, beautiful piece of that strategy. The question we want to ask is the one the keynote always skips: once the box is on your desk, what can it actually do, and what did you trade away to get there? Because the answer, when you chase it down with real numbers, is not the one on the slide.

02 / the physics

The bind nobody puts on a slide

Here is the single most important thing to understand, and it is the thing the marketing works hardest to keep quiet. When a language model writes you an answer, it is not really thinking hard in the way a benchmark implies. For every single token it produces, the machine has to stream the model's weights out of memory and back. Generation is not compute-bound. It is memory-bound. Which means the speed you actually feel, the words appearing on screen, is governed above all by one humble number: memory bandwidth.

And this is where consumer hardware in 2026 runs into a cruel little law of geometry. The boxes with enough memory to hold a big model are slow. The box that is genuinely fast cannot hold a big model. You can have capacity, or you can have speed. At a sane price, you cannot have both.

Look at the numbers and the bind jumps out. The RTX 5090, the fastest consumer card money can buy, moves data at a ferocious 1,792 GB/s. It will answer instantly. It also holds just 32GB, so it tops out at a well-quantized 32B model and physically cannot load the large ones. The boxes sold precisely for the large ones run at a fraction of that speed. The DGX Spark moves data at 273 GB/s. AMD Strix Halo, at 256. Even a Mac Studio that crested $10,000 manages 819. Reviewers landed on the cleanest way to describe the Spark: think of it as an RTX 5070 with 128GB of memory bolted on. Plenty of room, not much hurry.

Meanwhile, the place these models were actually born, the data center, runs memory at terabytes per second across whole racks of silicon wired together. We crawled inside one of those racks in our Vera Rubin NVL72 field guide, and the gap between that and a desk accessory is not the kind of thing you close with a clever purchase. It is a different order of machine. The keynote shows you the gold box. It does not show you this chart.

03 / the lived experience

What "slow" actually feels like

Bandwidth is an abstraction until you sit in front of it. So let us make it concrete, because this is the part you will feel every single day if you buy in.

On the DGX Spark, a dense Llama 70B model generates at about 2.7 tokens per second. Read that twice. That is slower than you can comfortably read. The Strix Halo does better, around 8 tokens per second on the same model, which crosses the line from painful into merely slow. The $10,000 Mac Studio runs the celebrated 671B DeepSeek model at roughly 17 to 18 tokens per second, genuinely usable, and a real achievement, but you paid Mac-Pro money and you still wait. And the RTX 5090 is instant, gloriously instant, right up until you ask it to load a model bigger than its 32GB, at which point it simply cannot.

Then there is the part that drives people quietly mad: the wait before the first word. Generation speed is one thing, but feed one of these boxes a long document and it has to read the whole thing before it can reply. On the Spark, that opening pause can stretch to two or three minutes on a long prompt. You ask a question, you go make tea, you come back, and it is still thinking about your question. The hosted model, by contrast, has read your prompt and started answering before your finger leaves the Enter key.

04 / the steelman

The part that surprised us: the models got good

We owe you the strongest possible version of the other side, because a year ago this argument was easy and now it is not. In early 2025 the open-weight models were visibly behind the closed frontier on almost every test. By the middle of 2026, on paper, that gap has very nearly closed.

A sparse model with 35 billion total parameters but only 3 billion firing per token, the kind of thing that fits on a high-end laptop in a 21GB download, now posts scores on real software-engineering benchmarks in the low-to-mid seventies, close enough to the closed flagships to make you blink. DeepSeek's open releases and Zhipu's GLM trade punches with GPT and Claude on the coding leaderboards. If you had told us two years ago that a model small enough to run at home would score like this, we would have laughed and ordered another coffee.

So here is the fair question pointed back at us: if the open models are this good, why do we still reach for Claude Opus 4.8 every single day? Two reasons, and neither one is a benchmark number.

A SWE-bench score is a single keystroke. Shipping software with an agent is ten thousand of them.

First, the open models that actually rival the frontier are not the ones that fit on your desk. DeepSeek V4 is a trillion-parameter mixture-of-experts model. Kimi, GLM at full size, the big Qwen variants: these are data-center residents, not desktop tenants. The open models small enough to run at home win the benchmark and then lose the things benchmarks do not measure: reliability across a forty-step agent run, tool use that does not quietly wander off the rails halfway through, long-context coherence that still holds at turn thirty, and the discipline not to confidently invent an answer when it does not know one. A leaderboard rewards a single hard problem solved once. Real work is the same model staying sane over thousands of steps in a row, and that is precisely where the polish of a fully tuned frontier system, model plus harness plus a year of hard-won reinforcement, still pulls clear.

Second, those scores deserve a raised eyebrow. Benchmark numbers depend heavily on the scaffolding around the model, so the same model with a different harness can swing ten points. Worse, the labs themselves have flagged training-data contamination creeping into the popular tests. A figure pulled straight from a vendor's own model card is a marketing multiplier until somebody independent reproduces it, and we treat it the way we treat every "up to" claim on a product page: interesting, unproven, and not a reason to spend four thousand dollars.

None of this makes the open models bad. They are remarkable, and for plenty of jobs they are exactly enough. It just means the daylight between "scores well on a chart" and "I would trust it to run my agents unattended" is still real, and the second one is the thing you are actually buying the box to do.

05 / the money

The math, and the inversion nobody saw coming

This is the part that should stop you with your card halfway out of your wallet. While you were saving up for the box, two prices moved in opposite directions, and almost nobody noticed.

Hardware got more expensive. The RTX 5090 launched at a $1,999 list price and now changes hands for well over $4,000, because the world ran short of the memory chips it needs. The DGX Spark launched at $3,999 last October and was quietly bumped to $4,699 in February for the same reason. The 512GB Mac Studio crested $10,000 and was then discontinued outright. The very silicon people are buying to "save money on AI" is in the middle of a price spike.

The frontier got cheaper. A year ago the top Claude model cost $15 and $75 per million input and output tokens. Today Opus 4.8 is $5 and $25, a two-thirds cut, for a model that is plainly more capable than the one it replaced. Sonnet sits at $3 and $15. Haiku at $1 and $5. DeepSeek will serve you tokens at fourteen cents a million. Renting the best AI on earth has never been cheaper, and the line keeps falling while the hardware line keeps climbing.

Now run the total cost honestly, not just the sticker. Every serious 2026 analysis we read lands in the same place: for the overwhelming majority of real workloads, the API is cheaper once you count everything. The breakeven against a frontier-class API sits somewhere north of 100 million tokens a month, a volume most production systems never touch, let alone a curious individual with a new toy. And the sticker is the small part of the bill. Self-hosting tends to cost three to five times the raw hardware price once you add the power, the cooling, the 1,200-watt power supply, and the sixty to a hundred and twenty hours a year of your own life spent patching, updating, and debugging a machine at two in the morning.

An idle GPU is not an asset humming with potential. It is a liability, billed by the depreciation schedule.

And depreciation is the quiet killer of the whole local dream. Your box is frozen the day it ships. The frontier is not. By the time you have amortized your Spark, several better Opus releases and a dozen sharper open models will have landed, and your hardware will be running the same weights it ran on day one, except now they are two generations behind. You did not buy a capability. You bought a snapshot, and snapshots age badly in a field that ships something better roughly once a month.

06 / the alternative

So here is what we actually do

We are not anti-local out of laziness or some loyalty to the cloud. We just put the four thousand dollars somewhere it compounds instead of somewhere it rusts. Here is the entire setup we run, and it costs roughly the price of two coffees a month.

A small box on Railway runs the orchestrator, a Node service and a Redis instance, on the Hobby plan. That is a five-dollar floor with five dollars of usage baked in, so a lean agent layer that mostly makes network calls barely touches it. On top sits an orchestration layer. We use LangGraph to hold the graph of agents and their state, though there are good alternatives if that is your taste: CrewAI, Temporal, or a framework of your own.

The clever part is the routing, and this is the bit people skip. Not every request needs the smartest model in the world. So we tier it. The cheap, high-volume work, classification, extraction, routing, first drafts, goes to Haiku or to a budget open model that costs cents. The everyday middle goes to Sonnet. Only the genuinely hard reasoning, the stuff that actually earns it, goes to Opus 4.8. One OpenRouter key gives a single OpenAI-compatible endpoint to three hundred models with automatic fallback, so if one provider hiccups, the request quietly reroutes instead of failing. A Claude API key sits alongside it for the calls we want to send straight to the source, where prompt caching shaves ninety percent off repeated context and the batch API halves the rest.

Memory and data live on Supabase's free tier: a real Postgres database with pgvector built in, so embeddings and relational rows sit together and the agents can actually remember. Free, right up until you are big enough that paying for it is a good problem to have.

That is the whole thing. Railway plus Claude plus OpenRouter plus Supabase. The frontier model, rented by the token. The cheap models doing the grunt work. A database that remembers. No box to cool, no weights to babysit, no resale value bleeding away on a shelf. And when a better model ships next month, which it will, we change one string and we are running it the same afternoon. That is the quiet superpower of renting: you are always on the newest thing, never the thing you bought in a moment of keynote enthusiasm.

07 / the honest other side

When the box is the right answer

We owe you the honest other side, because there are real reasons to put a model on your own hardware, and if one of these is you, buy the box without a shred of guilt.

Notice what is not on that list: "I want a private ChatGPT and I saw a really cool gold box." If that is the honest reason, and for most of the people rushing to buy in right now it quietly is, the box will let you down slowly, at two and a half tokens a second, while a far better model sits one API call away.

08 / the verdict

The verdict

The box is seductive because it feels like ownership, and ownership feels like the future. But strip away the leather-jacket glow and look at what you are actually buying: a depreciating asset that runs a weaker model more slowly, at a moment when the best model on earth is available to rent for the price of a few coffees a month, and getting cheaper while your hardware gets older and the chips inside it get more expensive to replace.

Local AI's moment is coming. Memory will get faster and cheaper, the small models will keep closing the gap, and one day the math will tip for ordinary people too. But here is the trap hidden inside waiting for that day by buying in early: when it finally arrives, Anthropic and OpenAI and DeepSeek and the rest will have sprinted again, and the frontier you can rent will be further ahead than ever. The target moves every month. Your hardware does not move at all.

So we will say it plainly, the way we would say it to a friend over coffee before they tapped "buy": unless your data cannot leave the building, or you are running a small AI factory, do not buy the silicon. Rent the frontier, route it cleverly, and put the four thousand dollars into something that does not lose a third of its value the day a better model ships. Which, lately, is roughly once a month.

If you do want to see the hardware the frontier actually runs on, we took an entire rack of it apart in our Vera Rubin NVL72 field guide. And if a card under the desk is still calling to you, our RTX 5090 review has the honest picture of exactly what 32GB does and does not buy you.

09 / the questions people actually ask

Questions people actually ask

Can I run Claude Opus 4.8 locally?

No. Anthropic does not release Claude's weights, so there is no file to download and nothing to run on your own hardware. The same is true of GPT-5 and Gemini. The frontier closed models live only behind a hosted API. What you can run at home are open-weight models like Qwen, Llama, DeepSeek's open releases, GLM and gpt-oss, and several of them are genuinely good. They are just not the same thing, and pretending otherwise is how people end up disappointed with an expensive box.

What is the best local LLM I can actually run at home?

On a 24GB consumer GPU, a well-quantized model around 32 billion parameters, Qwen 3.6 or Gemma 4, is the practical sweet spot and genuinely capable. On a 128GB unified-memory box, a DGX Spark, a Strix Halo system, or a large Mac, you can load 70B dense models or 120B mixture-of-experts models, though dense 70B generation is slow because of bandwidth. The trillion-parameter open models that actually rival the closed frontier need data-center hardware, full stop.

Is the NVIDIA DGX Spark worth it?

For the right job, yes. For a private ChatGPT, no. Its 128GB of unified memory lets you load models a 32GB card simply cannot, which is genuinely useful for fine-tuning and for serving large mixture-of-experts models. But its 273 GB/s bandwidth makes dense large models crawl, around 2.7 tokens a second on a 70B, and at roughly $4,699 it costs more than years of API usage for most people. Buy it for memory capacity and CUDA development, not for raw chat speed.

When does self-hosting an LLM beat the API on cost?

When your usage is high, steady and predictable. Independent 2026 analyses put the breakeven against a frontier-class API somewhere above 100 million tokens a month, and that is before you count power, cooling and maintenance, which tend to add three to five times the hardware price. Below that, and for anything bursty or experimental, the API wins on total cost almost every time. Privacy and compliance, of course, can justify self-hosting no matter what the spreadsheet says.

How much does a cloud-API agent stack cost to run per month?

Less than most people expect. A solo builder's orchestration layer on Railway's Hobby plan starts at a $5 floor, a Supabase database is free at small scale, and an OpenRouter key carries no subscription. The variable cost is tokens, and with tiered routing, cheap models for the bulk and Opus only for the hard parts, plus prompt caching and batch discounts, a serious personal project often runs in the tens of dollars a month. That is a small fraction of what the hardware to replace it would cost, and it is always running the newest model.