Local and private AI

Why it matters

Data is the asset AI consumes, and for an established business that data is often the most valuable and most regulated thing it holds. Sending it to a public service may be fine, may breach a client contract, or may create obligations under the Privacy Act around overseas disclosure that the business has not thought through. Private AI lets a business use models over its own documents, records and history with far less exposure. The trade is cost and complexity, and the mistake in both directions is common: businesses that over-engineer a private setup for data that was never sensitive, and businesses that paste client files into a consumer chat tool because private options seemed too hard.

What "private" actually means

The word covers several different arrangements, and vendors blur them. A public AI service on an enterprise plan with a no-training clause and Australian data residency is more private than a free tier, but your data still travels to and is processed by someone else’s system. A private cloud deployment runs the model in an isolated environment that only your business uses, often in an Australian data centre, under a contract that specifies where data lives and who can see it. A local deployment runs the model on hardware you own or lease, in your premises or a co-location facility, and the data never leaves your network.

Each step up adds control and removes convenience. The right level depends on the sensitivity of the data, your contractual and regulatory obligations, and your capacity to run infrastructure. Few small businesses need the top tier for everything; many need it for one specific workload, such as searching confidential client files, while general writing assistance stays on a public business plan.

Open-weight models and what they can do

Local and private AI is possible because a number of capable models are published with their weights, the trained parameters, available to download and run. Several major technology companies and research groups release them under licences that allow commercial use, though licence terms vary and should be read. These models run on a workstation or server with suitable graphics hardware, or in a private cloud environment, and are close enough to the leading public models for most business tasks such as summarising, extracting, classifying and answering questions over your documents.

They are generally smaller and less capable at the hardest reasoning tasks than the largest hosted models, and they do not improve unless you update them. For the typical private workload, which is answering questions from a body of internal documents, the gap is often unimportant, because the quality of the answer depends more on the retrieval setup than on raw model size. Retrieval-augmented generation, where the system finds the relevant passages from your documents and gives them to the model to answer from, is the workhorse pattern of private AI.

Costs and skills, honestly

A local setup capable of serving a small team starts with a server or workstation with capable graphics hardware, which is a meaningful capital cost rather than a monthly subscription, plus power, backup, security and someone to maintain it. A private cloud tenancy avoids the hardware but costs considerably more per month than a public business plan, because you are paying for dedicated capacity whether or not it is used. Both need someone, in-house or contracted, who can deploy and update models, build and tune the retrieval layer, manage access, and keep the whole thing patched.

The hidden costs are the same as for any infrastructure: the model and software need updating, the document index needs refreshing as files change, and the system needs monitoring so that failures are noticed. A business that builds a private AI system and then leaves it is running an unpatched server holding its most sensitive data, which is worse than the problem it set out to solve. Budget for ongoing operation from the start, whether through an internal owner or a managed service.

Deciding without over-engineering

Start by classifying the data each AI use case would touch. Public information and generic drafting need no special treatment. Internal business information, such as procedures and pricing, is usually fine on a public business plan with a no-training clause. Personal information about customers and staff needs a plan that meets your Privacy Act obligations, which a business-grade public service with Australian residency may satisfy. Highly sensitive material, such as health records, legal files, financial records under professional obligations, or anything a contract says may not leave Australia or your control, is where private cloud or local AI earns its cost.

Then match the level of control to the highest classification in each use case, not to the whole business. A law firm might run document search over client files locally, and still use a public assistant for marketing copy. This is where specialist help is valuable: SMBH Consulting’s private AI work typically begins with exactly this classification exercise, because it determines whether a business needs a server, a private tenancy, or simply a better-configured public plan.

How to approach private AI

1. Classify your data Public, internal, personal, highly sensitive. Note contractual and regulatory restrictions.
2. Map AI use cases to data classes Most will be fine on a public business plan; find the few that are not.
3. Check what a public business plan offers Australian residency, no training, retention controls and a contract may be enough.
4. Scope the private workload narrowly Usually document search and summarisation over a defined set of files.
5. Choose private cloud or local Based on sensitivity, volume, budget and who will run it.
6. Pilot with a subset of documents and users Measure answer quality and retrieval accuracy against known questions.
7. Plan operations before launch Patching, index refresh, access reviews, monitoring and an owner.

Real-world example

A Launceston accounting practice searches its files without sending them anywhere

Tamar Valley Accountants has twelve staff and twenty years of client files: tax returns, financial statements, correspondence and advice. Partners want the ability to ask questions across a client’s history and to summarise long ATO correspondence, but professional obligations and client engagement terms make them unwilling to upload client files to a public service. They classify their data and find that only the client-file search truly needs isolation; email drafting and marketing can stay on a public business plan.

Rather than buying a server, they deploy an open-weight model and a retrieval layer in an isolated tenancy at an Australian data centre through a managed provider, who patches and monitors it. Access mirrors their existing file permissions, so staff can only query clients they already work on, and every query is logged. The pilot indexes one partner’s clients first and tests the system against fifty questions with known answers; retrieval tuning takes several weeks before accuracy satisfies them. The ongoing cost is several times their public AI subscriptions, which the partners accept for the specific workload because it lets them use AI on the files that matter without changing their client terms or their professional risk.

Common mistakes

• Going private for everything Most AI use in a business touches nothing sensitive. Isolate the workloads that need it and keep the rest simple.
• Confusing a no-training clause with privacy A business plan that does not train on your data still processes it elsewhere. Check residency, retention and access, not just training.
• Underestimating operations A private AI server that nobody patches or monitors is a liability holding your most sensitive data.
• Expecting public-model quality from small local models Open-weight models are excellent for retrieval over your documents and weaker at the hardest reasoning. Design for what they do well.
• Skipping access controls A system that lets any staff member query any file has created a new data breach risk inside the firewall.

KPIs to track

• Retrieval accuracy Share of test questions where the system finds the right source passages. Measure against a set of known answers before launch.
• Sensitive data outside approved environments Incidents of restricted data entering a public tool. Target zero.
• System uptime and patch currency Availability, and days since the last security update. Treat it like any other production server.
• Cost per active user per month Total private AI cost, including people and hardware, divided by users. Compare with the risk it removes.
• Access review completed Quarterly confirmation that who can query what still matches roles.

Frequently asked questions

Is local AI as good as the big public models?

For summarising, extracting and answering questions over your own documents, open-weight models are usually good enough and the retrieval setup matters more. For the hardest reasoning and the latest capabilities, large hosted models lead.

Can I run AI on an ordinary office computer?

Small models run on a modern laptop for individual use. Serving a team with acceptable speed over a large document set needs a server or workstation with capable graphics hardware, or a private cloud tenancy.

Does private AI satisfy the Privacy Act on its own?

No. It helps with overseas disclosure and third-party access, but your obligations around collection, security, access, retention and notification of breaches still apply, and a poorly secured local system can breach them.

What is retrieval-augmented generation?

A pattern where the system first searches your documents for passages relevant to a question, then gives those passages to the model to answer from. It grounds answers in your material and sharply reduces invented facts.

Key takeaways

• Private AI is a spectrum from business-grade public plans to fully local systems; match the level to the data, not to anxiety.
• Classify data first and isolate only the workloads that genuinely need it.
• Open-weight models plus retrieval over your documents handle most private workloads well.
• Budget for operations: patching, index refresh, access reviews and monitoring are the real ongoing cost.
• Private infrastructure eases overseas-disclosure concerns but does not remove your other Privacy Act obligations.