Why We Abandoned the Biggest LLM for Four Smaller Ones
I thought bigger was better. I was wrong.
When my team at Soliant Consulting set out to build a local AI assistant for a large community college district, working alongside Apple, my first instinct was to find the most powerful open-source model available and hand it everything. That logic made sense to me at the time. More parameters, more capability, better answers. Plus, it saved me from a lengthy stretch of model evaluation work; just pick the biggest most capable model, right?
What I got instead was a slow system burning expensive compute on tasks that a much smaller model handles in a fraction of the time.
The Assumption That Slows Everyone Down
The model I started with was a 120 billion parameter open-source model. My thinking: if it is the biggest one I can get, it will just work. And technically, it did work. The problem was, it was slow.
The bigger the model, the slower it runs. That is not a flaw. It is physics. More parameters mean more compute per inference. When we were building a system where a staff member types a question and expects an answer in a few seconds, waiting 10 seconds just for intent classification is not acceptable. The user experience falls apart before the system has even started looking for the answer.
There was a second problem. This system had two very different functional domains: device inventory questions and HR policy document questions. A question about replacing 10,000 MacBooks has nothing in common with a question about what a union contract says about overtime. Asking one model to handle both domains well, and fast, is asking too much.
I found that to not be true in practice. And once I accepted that, everything about the architecture changed.
What Smaller Models Are Actually Good At
A small model asked to do one narrow thing, and only that thing, does it fast and does it well.
Intent classification is the clearest example. We needed the system to determine, very early in each request, whether a user’s question was about device data, a policy document, or a general conversational question. That is a three-way classification. It is not complicated. But if you hand it to a 120 billion parameter model, you are waiting 10 seconds for a task a small model solves in about one second.
Smaller models are genuinely fast at narrow tasks. And because we only ask each model to do one thing, we are not sacrificing quality. A model that is not capable of generating rich, detailed responses is perfectly capable of deciding: this is a device question, route it accordingly. That is all we are asking it to do.
This is the insight that changed how we approached the whole system. You do not need one model that is good at everything. You need several models, each right-sized for exactly one job.
How We Split the Work Across Four Models
The production system my team and I built runs four models. Each has one job.
- A planning model (Qwen3 8B) classifies intent and decides which tool to invoke. It runs in about one second.
- An execution model (Qwen2.5 Coder 32B) generates SQL queries from natural language questions. This is the largest model in the stack because writing correct SQL against a complex schema genuinely requires more reasoning. It runs in about two seconds.
- A response model (LLaMA xLAM 8B) takes raw output from the execution stage and formats it into a readable, conversational answer. One second.
- An embedding model (all-MiniLM-L6-v2) converts policy documents into vectors for semantic search.
Total response time for most queries: two to four seconds.
The largest model in this stack is 32 billion parameters. The smallest is 8 billion. Neither is close to the 120 billion parameter model I started with. The system is faster, cheaper to run, and easier to reason about than the single-model version ever was.
One more thing worth saying: this architecture is model-agnostic. The models listed above reflect where the open-source space was when we built this. A better planning model releases next month? Swap it in. The system is designed for that.
The Question to Ask Before You Pick a Model
The teams I see struggle most with AI performance start with the same assumption I did: bigger model, better results. It does not hold up.
A model that is too large for its task is just slow. A model that is too small for a task it was not designed for produces poor output. Neither is a model problem. Both are design problems, and we treat them as such.
The question to ask before choosing a model is not “how big is it?” It is “what is the one thing I need this model to do, and what is the smallest model that does that well?”
For intent classification and response formatting, the answer is usually a fast 7–8 billion parameter model. For generating SQL queries against a complex schema, you need something larger. For embedding documents, you need a model that produces good semantic representations, not one that generates text at all.
Get those assignments right, and you can build something that runs at two to four seconds per query on local hardware with no cloud dependency. Get them wrong, and you spend a lot of time waiting.
Right-Sizing AI Applications with an Experienced Team
Right-sizing an AI application takes hands-on experience with how different models perform across different tasks. The open-source model space also moves fast. What was the right choice six months ago may not be the right choice today.
My team and I at Soliant Consulting stay current on this as part of how we work. We design AI systems that match the actual requirements of each task, run efficiently, and are easy to update as better models become available. That is what long-term thinking about AI looks like.
Schedule a free 30-minute consultation with our AI development team. We will look at your specific use case, identify where AI adds genuine value, and help you avoid the bigger-model trap.
