The most revealing metric for local AI is not printed boldly on the box. It is called memory bandwidth, it is measured in gigabytes per second, and it decides whether a model runs smoothly on your machine or stutters slowly to death. Anyone serious about experimenting with AI who leans on the TOPS and NPU figures is looking in the wrong place. Compute has grown cheap. Bandwidth is the scarce resource.
That sounds counter-intuitive, and it is. Which is why it pays to look closely — not at benchmark leaderboards, but at what actually happens inside the chassis as a model composes token by token.
Why bandwidth wins
Language models work autoregressively. Each next word appears only after the full weight of everything so far has streamed through memory once. The multiplication units are not waiting on heavy mathematics — they are waiting on memory. Understand that, and you understand why a GPU with little VRAM is useless against a large model no matter how fast its shaders clock: the model must fit and flow through, every step.
As a rough guide: tokens per second approximate memory bandwidth divided by model size. A seventy-billion-parameter model in 4-bit quantisation occupies nearly forty gigabytes. Across a 273 GB/s pipe that is, in theory, around seven tokens per second — minus overhead, minus KV-cache, minus whatever the operating system is doing beside it. Across ordinary laptop DDR at 64 GB/s the same calculation becomes an exercise in patience. Both times the compute cores sit idle. The difference lies solely in how wide the pipe is.
That is precisely why certain chips have lately grown interesting that nobody would have put on an “AI accelerator” shortlist.
Two chips nobody had on the list
Consider, first, the Apple M4 Pro. Apple specifies roughly 273 GB/s of unified-memory bandwidth — shared storage for CPU and GPU alike, configurable up to 48 gigabytes. Nothing about it is a data-centre accelerator. But because the memory is so broadly attached and equally visible to CPU and graphics unit, mid-sized models run with surprising fluidity. That is not marketing magic; it is physics.
Second, the AMD Ryzen AI Max+ 395, codename Strix Halo. AMD specifies up to 256 GB/s, likewise unified, and the platform holds up to 128 gigabytes of LPDDR5X. You find it in compact mini-PCs, workstation laptops, and devices like the Framework Desktop. Again: not an H100, not a miracle machine. But anyone able to lay a large model entirely into fast memory while a competing card suffocates in out-of-core swapping suddenly achieves realistic throughputs for money that elsewhere buys a fraction of a professional board.
Honesty is compulsory here. Neither chip is magic. Thermal throttling under sustained load is real, particularly in slim enclosures. Driver and runtime maturity differs: Apple’s Metal path is smooth; AMD’s ROCm/OpenCL route has improved but remains rougher than the CUDA world everything is measured against. And of course a single desktop chip trails by miles what an H200 or B200 card delivers in a rack. Claims that consumer hardware is “just as good” are nonsense. Claims that it is irrelevant are nonsense too.
One dry observation: benchmark charts lie routinely, because they measure time-to-first-token on a warm cache — not Tuesday afternoon, six tabs open, half the model swapped out. Plan for AI under load, not under laboratory conditions.
Phase one: Click and marvel
The natural entry point runs through desktop applications. Ollama pulls a GGUF; LM Studio offers a tidy interface for browsing models, comparing quantisations, trying things out. That is valuable. It is the quickest route to feeling how an 8B, 13B, or 32B model behaves, where the quality curve bends, where 4-bit remains acceptable and where it does not.
This phase is playful, and it ought to be. Anyone shouting “architecture!” at this stage misses the point: you must feel what the tools do before you formalise them. But the limits show themselves soon enough. Everything is tied to one machine, one user, one graphical session. The moment a second person wants the model, the moment a nightly script should infer automatically, the GUI becomes an obstacle. Network access feels bolted on, not designed.
Phase two: The desk becomes a server
At some point that no longer suffices. You want one endpoint the whole office, the whole home network, every automation can reach. You bind Ollama or a lightweight runner to ::, put OpenWebUI in front of it — multiple users, authentication, model selection, document chat — and point tools like opencode, the CLI coding agent, at the local OpenAI-compatible endpoint.
Suddenly the workstation is infrastructure. Every editor, every colleague, every pipeline speaks to the same address. That is a genuine paradigm shift even though it looks technically small: an application has become a service. And with the service come the duties familiar from infrastructure — transport encryption, authentication, logging, updates, model governance. Whoever takes this step has stopped tinkering and begun operating. Many notice only when someone else accesses the model for the first time.
Phase three: When it stops being a toy
Growing ambition brings the next tier of tools. vLLM is the engine that made GPU inference economic — continuous batching, PagedAttention, high throughput, clean handling of many simultaneous requests. LiteLLM sits ahead of it as a proxy and router: one uniform, OpenAI-shaped API across many backends, key management, fallback paths, budget routing, integration with metrics and logging. And Bifrost is the kind of project you meet once you start pooling and sharing GPU capacity across workloads or tenants instead of dedicating a card to a single process — utilisation over hero boxes.
These tools reward bigger hardware: multiple GPUs per node, NVLinks, a rack rather than a desk. But they also punish negligence harder. Continuous batching helps only when enough requests arrive simultaneously — for a lone power user the overhead never pays back. A router is worthless with only a sluggish backend behind it. Pooling only works when fairness, queue depth, and cold starts are managed seriously. In short: at this point experimentation ends and distributed systems begin. Monitoring, scheduling, cost attribution, update strategies — all topics known from classical IT operations, now unexpectedly part of AI.
Many homelabbers hit this wall and choose one of two directions: retreat to a comfortable single box that is good enough — or consciously step into proper operations. Both legitimate. But believing a further click will carry you past it is a mistake.
The difficult art: thinking hardware and software together
Here lies the real difficulty. Neither hardware alone nor software alone wins. Pairing is everything.
Buy a huge GPU and drive it through a single-streamed runner without batching and you squander ninety percent of its capability. Stand up vLLM for a workplace with a single user and you pay overhead with no return. Crush a large model into 4-bit where 8-bit was qualitatively necessary and you save memory and lose precision. Conversely, run everything in FP16 because “higher sounds better,” fit less in VRAM, and freeze up. Quantisation is not a switch but a dial. Cooling, noise, and steady-state power consumption decide whether a device may live in the living room or belongs in the basement lab. Depreciation belongs in the calculation, not in the footnote.
And then the elephant nobody likes to acknowledge: API pricing. It looks cheap — cents per million tokens — until volume escalates. Agentic loops multiply requests tenfold, a hundredfold. Retry storms, redundant retrieval passes, verbose system prompts reshipped every turn, a wrongly sized model for a trivial task: a cheap experiment turns into a four-figure monthly invoice. Method determines price more than tariff.
Self-hosting inverts this: fixed cost instead of variable. Somewhere it breaks even economically, but only at respectable utilisation. A GPU idling round the clock combines the disadvantages of both worlds — capital outlay and opportunity cost. Romance is no use here. It takes arithmetic, and the honesty to assess your own utilisation realistically rather than idealising it.
A reason to celebrate
Through all the effort, there is cause to celebrate, and it carries weight: the entire ladder described above is open source. Ollama, llama.cpp, vLLM, LiteLLM, Bifrost, OpenWebUI, opencode — community-built, inspectable, auditable, forkable. Nobody can take them from you, nobody can rewrite the terms overnight without leaving you the option to stand still.
One nuance belongs here: LM Studio is convenient, but it is not free in the OSI sense. It is gratis for personal use, commercially licensed otherwise. Those who value consistently free software choose the Ollama and llama.cpp path. That is not dogma; it is a clarification worth knowing before building architectures atop it.
Seen more broadly: AI has shaken our professional lives — all of them, in some fashion. It arrived, and it stays; of that I am certain. Denying it is nostalgia. But naïveté is the precise opposite and equally wrong. Convenience conceals dependence, and it conceals costs that rise quietly. The knowledgeable stance is: use AI deliberately, on your own terms, understanding what it costs, what it can do, and where it fails. That is not a stance against the technology. It is a stance for your own agency.
What this has to do with libcom.de
For a quarter-century I — Jochen Demmer, libcom.de — have built on open-source infrastructure. Increasingly in recent years that includes operating AI stacks on owned hardware: for log analysis, drafting, code review, internal assistant systems, document-grounded search. Not as demos, but in daily practice and on customer systems.
From that practice come judgements no tutorial teaches: where the break-even between API and self-operation tips, how the quantisation-versus-quality dial stands for which workload, what network and security architecture a cleanly exposed enterprise inference endpoint requires — and where the boundary lies between a homelab toy and a maintained service. The commonest task is not procuring hardware but pairing the right hardware with the right serving strategy, rather than buying an expensive box first and hunting for software to justify it afterwards.
If you wonder whether local AI makes sense for your environment, which hardware suits your planned software, and what a stack looks like that does not collapse after three months: write to contact@libcom.de. We take an honest inventory of what should run where you are — openly, without sales pressure, with a view to what lasts long-term.
It begins with curiosity and ends, for whoever takes it seriously, with infrastructure. Between the two lies the finest craft presently being reinvented — piece by piece, on your own hardware, with open tools.
Note: This article offers general orientation on hardware and software choices as well as cost considerations and replaces neither procurement nor legal advice. Performance and pricing figures are illustrative and may vary with configuration, driver situation, and market developments; all details are provided without warranty.