How to Run a Full AI Assistant On-Premises with No Cloud Dependency
“No data leaves our building.” That was the starting point for everything my team and I at Soliant Consulting built.
A large community college district, working with Apple and Claris, needed an AI assistant that lets staff query device inventory, calculate total cost of ownership, and search HR policy documents in plain English. The requirement was non-negotiable: no OpenAI, no Anthropic, no AWS. Not even a private cloud instance. Everything had to run on hardware they own, in a building they control.
Three years ago, that requirement would have been a polite way of saying the organization was not ready for AI. Today it is buildable. And the hardware that made it possible was already sitting on desks across the district.
The Privacy Requirement That Shaped Every Design Decision
The concern was real and specific. This district manages HR records, salary schedules, union contracts, and device inventory for thousands of employees and students. That data is sensitive. It is subject to compliance requirements. And it is exactly the kind of data you need to give an AI assistant context to answer questions well.
When a staff member asks an AI assistant about HR policy, the question and the relevant document chunks travel to whatever server is running the inference. With a cloud-based model, that means your data leaves your network. For many organizations, that is an acceptable tradeoff. For this client, it was not.
We had done hybrid approaches on other projects, where inference runs on a private AWS instance, and data stays within a controlled environment. That was not acceptable here either. The requirement was absolute. The models, the data, and every step of inference had to run on hardware the district owns and operates.
That constraint did not limit the system. It defined it.
Why Apple Silicon Turned Out to Be the Right Hardware
Apple did not set out to build AI inference hardware. But the Mac Studio turned out to be genuinely well-suited for it.
The reason comes down to unified memory architecture. A Mac Studio with a high-end Apple Silicon chip has a large pool of unified memory shared between the CPU and GPU. Running a large language model locally requires keeping the model’s parameters in memory during inference. The more memory you have, and the faster it is accessible to the GPU, the faster the model runs.
Apple backed into this. They built high-quality machines with powerful GPUs and large unified memory pools for entirely different reasons, and those design decisions line up almost perfectly with what local AI inference needs. NVIDIA gets most of the attention in AI hardware conversations. Apple Silicon is a legitimate option that most people in this space underestimate.
We ran four concurrent models on this hardware, including a 32 billion parameter SQL generation model, at two to four second response times per query. For a staff-facing application, that is fast enough to feel responsive.
What the Full Local Stack Actually Looks Like
Building a local AI application with no cloud dependency means replacing every cloud service with a local equivalent. We built the following stack, entirely on-premises:
- LM Studio runs on the Mac Studio hardware and serves all four AI models locally. No calls to OpenAI, Bedrock, or any external inference API.
- A Node.js backend API orchestrates every request, routes questions to the right tool, manages conversation history, and syncs device data from FileMaker Server.
- A Python FastAPI RAG service handles document ingestion and semantic search. Staff upload policy documents through a React UI; the service processes, chunks, and embeds them into ChromaDB.
- ChromaDB is the vector database that stores document embeddings for semantic search.
- SQLite stores device data synced from FileMaker Server and full conversation history.
- All five services run in Podman containers on the Mac Studio, Docker-compatible and rootless.
The chat interface is a React application embedded inside a FileMaker WebViewer. The front end lives inside FileMaker. Everything behind it runs locally on the same hardware.
Deployment is scripted. A single shell script handles the git pull, container rebuild, dependency checks, and service startup. Data persists across deployments through named volumes. The system is straightforward to maintain once it is running.
Local AI Is Not a Compromise
The most important thing my team and I at Soliant Consulting took away from this project is that local AI is not a fallback for organizations that cannot use the cloud. It is a legitimate architectural choice with real advantages.
The obvious advantage is privacy. Every document, every query, every response stays on-premises. There is no data residency question. There is no vendor terms-of-service risk. There is no external API outage that takes your application down.
But the less obvious advantages matter too:
- Local inference has no per-token cost. The system runs the same whether ten staff members are using it or a hundred.
- The organization decides when and whether to update the models. Nothing gets deprecated without your knowledge.
- Response times are competitive. Two to four seconds per query on local hardware, with models that top out at 32 billion parameters, is fast enough for a production staff-facing application.
- There is no vendor-lock-in that becomes a vulnerability, as we have seen with the recent Anthropic Fable issue, where outside forces can prevent access to your chosen model
The tradeoff is real. Local hardware requires upfront investment. The open-source models available locally are not the largest commercial models. But for well-scoped applications where the queries are known, the domains are defined, and the data is sensitive, the capability gap is smaller than most people expect.
Organizations that have been waiting for local AI to be ready should stop waiting. The hardware is there. The models are there. The tooling is there. Soliant Consulting has built it, and the lessons are directly transferable.
Your Partner for Local AI Applications
My team builds this kind of system end to end. The architecture is documented, the tooling is proven, and the hard lessons about model selection, data privacy, and local inference performance are directly transferable to other organizations. Building a local AI application is not a research project anymore. It is an engineering project.
Contact us to chat with our AI development team. We will walk through your data privacy requirements, identify the right hardware and model configuration for your use case, and give you an honest assessment of what local AI can and cannot do for your organization.
