I use AI heavily for development work at TNG. Between commercial providers and models hosted on TNG’s own GPU cluster, I have access to much stronger models than I can realistically run on my laptop. Even so, local models still matter to me.
For most tasks, especially agentic coding, hosted frontier models are still very much in a different class. But local models have also become useful in different ways: for experimentation, for offline use, and for a small but growing set of workflows where they are actually good enough. For a long time, though, I mostly just saw them as a way to learn.
Hardware sets the limits
In late 2024, it was time to replace my work laptop at TNG. When choosing the new one, I deliberately tried to get as much VRAM as reasonably possible. At the time, that was mostly future-proofing. I thought I might want to play around with AI or ML locally at some point, and I did not want the hardware to rule that out before I had even started.
In hindsight, that decision proved more important than I had expected. The machine I chose has 8 GB of VRAM. This set the limits of what I could run locally. It is enough to be interesting but tight enough that every choice around model size and quantisation counts. Without it, most of the experiments in this post would never have happened.
RAG, embeddings, and Ollama
My first real push into local LLMs came through a RAG workshop at work. That was where I started trying things out locally with Ollama, both for embeddings and for chat models. At that stage, I was mostly just trying to understand the basics: how to run models locally, how embeddings can be used to get more grounded answers, and what kinds of models were even small enough to be practical on my machine.
That led me to download a growing collection of models. On the chat side, I tried small Llama 3.1 and 3.2 variants. On the embeddings side, I looked at models like BGE and whatever else seemed promising on leaderboards such as MTEB while still being runnable locally. Ollama was a very good fit for that phase. It removed almost all friction. I just needed a way to get started, without a lot of control, and that is exactly what Ollama gave me.
That early phase also turned out to be useful in a practical way. Later, in a customer project, we worked on semantic search with Qdrant and OpenAI’s text-embedding-3-large. The experience and intuition I gained by playing around with local embedding models helped me quite a bit with getting things right in the project and also allowed me to do some early prototypes locally.
Trying local models for coding
Naturally, I also tried to use local models for coding. At the time, that mostly meant small Qwen2.5 and Qwen2.5 Coder models (0.5B–7B), often through Continue.dev to get AI autocomplete. In principle, that worked. I could run the smaller variants fully in VRAM. Slightly larger ones (14B) still ran, but slowly. But once I got into the bigger models, it stopped being practical very quickly.
Things worked, just not well enough to actually use it. Autocomplete in particular never convinced me. While I could get a local model to produce completions, it was often too slow or too weak to feel worth the friction. And even with stronger hosted models, I am just not a huge fan of AI autocomplete. It tends to get in the way more often than it helps. Continue.dev was mostly a crutch at the time, since there was nothing better. The project is no longer maintained either; the team was acquired by Cursor.
From autocomplete to agents
My use of AI moved through a few different phases: autocomplete first, then chat, then agentic workflows. Each shift changed what I needed from models and tools. Instead of inline suggestions, I was now after models that could reason, edit, plan, and stay coherent over a longer interaction.
That shift also carried over to my tools. I began using Zed more as a VS Code alternative, partly because it already had better built-in AI support and worked with both TNG-hosted and local models for chat and agentic workflows. I still use both editors, but mostly just as editors. For agentic work, OpenCode has become my default choice.
When local started looking plausible
There was no single point in time when local models suddenly became good. Instead, there was just a slow but steady increase in what is reasonably possible with them.
Early small models were useful mostly as learning tools. They made the ecosystem accessible, but they did not make local inference compelling for serious work. Qwen2.5 and Qwen2.5 Coder were among the first local models where I thought: not there yet, but I can see where this is going.
Later Qwen models pushed that further: first Qwen3 and Qwen3 Coder Next, and more recently Qwen3.5 and Qwen3.6. The newer MoE models in particular changed my expectations. Models in the rough 30-something-billion-parameter range had been runnable before. Now, they were actually really useful.
Other families interested me for different reasons. Earlier on, Gemma 3 stood out to me mostly for good prose, multilingual output, and vision, but not so much for agentic work. Gemma 4 changed that to some degree. It feels much more competitive for agentic workflows, while still keeping a lot of the writing-quality strengths that made Gemma attractive in the first place. GLM 4.7 Flash was another strong indicator that local MoE models were getting serious. It was one of the models that made me think: this is something I might actually want to keep around and tune properly, to get the most out of it.
Then there were the models that stuck with me simply because they stretched my sense of what was possible. gpt-oss-120b belongs in that category for me. I was genuinely surprised that I could run it at all. Relatively slow, and not really practical for typical work, but the fact that it ran was exciting.
By that point, local models were still clearly inferior to hosted options, but they had stopped feeling like a toy problem.
Not everything that was promising on paper worked well for me in practice, though. Every model release typically comes with claims about very good benchmark results. But not all of that holds up in practice. Devstral 2, for example, never really clicked for me. I suppose part of taking local models seriously is learning which ones sound exciting and which ones actually fit your hardware and your use cases.
The tools changed too
As the models changed, my tooling changed with them. Ollama was the right place to start because it made local inference almost trivial. Later, LM Studio became a better fit for how I was using local models. It gave me more convenience for trying many models, comparing settings, and interacting with them in a more exploratory way, without forcing me all the way down to raw inference tooling.
But at some point, even that middle ground started to feel limiting. I found myself adjusting model settings manually in a GUI, comparing results in an ad hoc way, and increasingly wanting more control and more reproducibility. I began thinking more seriously about moving to raw llama.cpp, but for a long time I was not ready to give up the ease of use of LM Studio.
That has changed now. I want more structure, and I have reached the point where convenience matters less than control. I needed a setup that I could script, benchmark, and understand properly. And llama.cpp was the first tool that felt right for approaching local experimentation more systematically.
What local models are good at
Local models have not replaced hosted ones for me. For coding in particular, hosted frontier models are still clearly stronger.
The more interesting question is what local models are useful for on their own terms.
Running models locally teaches me things I do not learn by only calling an API. I can try models, quants, context settings, and odd ideas far more freely when everything runs on my own machine. And I care about offline and fully local workflows, even though I am not actively using all of these yet: image tagging in a personal photo library, a fully local assistant in Home Assistant, local speech workflows, or simply being able to work on a plane without depending on a network connection.
Access to strong hosted models makes local less necessary, but not less worth understanding.
Where I am now
Right now, I am still experimenting, but in a more structured way than before. The question has shifted from “can I run this locally?” to “which models and quants are actually worth using on my hardware, and under which trade-offs?”
At the moment, the most interesting local candidates for me are from the Qwen family, Gemma 4, GLM 4.7 Flash, and in some cases gpt-oss. I am now focused on the practical questions around them: context versus speed, dense versus MoE, and which quants make sense on my laptop instead of just on paper.
That is why I have now made the jump to raw llama.cpp and started benchmarking more systematically. Hosted models are still the obvious default for much of my work, but local models have reached a level where I take them seriously as a practical option rather than just a hobby.
In a follow-up post, I will focus on that: the hardware constraints, the benchmark setup, the trade-offs between speed, context, and quality, and which local models and quantisations currently look most promising on my machine.