On-prem vs cloud AI: a decision framework for Australian boards

Most arguments about running AI in the cloud or on your own hardware start with the technology and end in a stalemate. That’s the wrong place to start. Begin with your data instead: what it is, who’s allowed to see it, and what happens if it leaves the building.

The stalemate has a predictable shape, and if you’ve sat through the meeting you’ll recognise it. One side argues cloud, because the strongest models live there and nobody wants to explain a server purchase. The other argues on-prem, because someone has read about data sovereignty and the word “breach” has been said aloud. Both positions are really instincts wearing arguments, and neither can win because the question as framed, cloud or on-prem, has no answer. It’s like asking whether the company’s documents should be stored in the office or in a facility: it depends entirely on which documents.

Start with the data, not the model

Sort your information into three piles. There’s data you’d happily hand to a public system. There’s data that’s commercially sensitive. And there’s data you’re legally or contractually bound to keep under your own control. When boards actually do this, the third pile usually turns out smaller than they feared, which means a hybrid setup is on the table rather than an all-or-nothing call.

Do the sort with specifics, not categories. Marketing copy, public tenders, published financials: first pile, no drama. Internal pricing, customer lists, board papers, unreleased strategy: second pile, where the question becomes which contractual and technical protections you’d accept. Client-privileged material, health records, anything under a government or defence contract with data-handling clauses, anything covered by the Privacy Act where the consequences of offshore disclosure are serious: third pile, where the answer needs to be verifiable rather than promised. The exercise takes a workshop, not a quarter, and it converts an ideological argument into a routing table. Most organisations discover something like 70% of their AI-relevant work touches only the first two piles.

While you’re at it, note who actually asked for what. A lot of on-prem enthusiasm evaporates when the board sees that the use cases on the table are drafting proposals and summarising meeting notes, and none of them touch the third pile at all.

Weigh three forces honestly

Sovereignty is about where the data has to physically and legally sit, and it deserves precision rather than vibes. “Australian region of a US cloud provider” and “hardware in our own comms room” are different sovereignty postures, and different again from “vendor promises in a contract”. For some obligations, an Australian cloud region with the right contractual terms satisfies everyone who needs satisfying. For others, a client agreement or a regulator’s expectations mean the honest reading is that the data doesn’t leave infrastructure you control. Get your actual obligations in writing from whoever advises you on them, because we’ve seen boards build bunkers against threats no contract required, and others wave sensitive data offshore because nobody read the client agreement closely.

Cost cuts the other way depending on the option. Cloud is cheap to start and predictable as you scale, which makes it unbeatable for experiments and spiky workloads; you pay for what you use and can stop any time. But steady, heavy usage inverts the maths. An organisation running large volumes of documents through cloud models every day can watch the monthly bill climb to where a one-off hardware spend, and capable open-weight models are far less demanding than they were, pays for itself inside a year or two. On-prem trades a bigger upfront spend for a known long-run cost you actually own. The honest comparison needs your projected volume, not this month’s pilot usage, and it should include the operational cost of running your own kit, because a box someone has to patch and monitor is never free.

Then there’s capability. The strongest models are easiest to reach in the cloud, and for the hardest reasoning tasks that edge is real. But plenty of capable models now run fine on modest local hardware, and here’s the part vendors won’t lead with: most business workloads don’t need the frontier. Extracting fields from invoices, summarising reports, answering questions over your own documents, drafting routine correspondence, a well-set-up local model handles all of it. The capability question isn’t “which model is best?” It’s “is the model in this deployment good enough for this specific job?”, and for the third-pile workloads the answer is usually yes.

The hybrid default, and how it evolves

Put the three forces together and most Australian organisations land in the same sensible place: cloud models for the first two piles, where capability and convenience win, and a private deployment for the third pile, where verifiable control wins. The two coexist fine. What makes it work is the routing discipline, clear rules about which work goes where, written into the same one-page policy that governs the rest of your AI use.

This decision is rarely permanent, and treating it as reversible changes how you make it. A lot of our work starts in the cloud to prove the use case quickly, cheap, fast, no procurement of hardware, then moves the sensitive workloads on-prem once it’s clear the thing is worth keeping and the volumes justify it. Build with that portability in mind from day one, keeping the prompts, the document pipelines and the evaluation criteria separate from any one provider’s plumbing, and the later move is a project measured in weeks. Build carelessly against one vendor’s proprietary stack and the move becomes a rebuild, which is the same lock-in trap software buyers have been walking into for decades, wearing a new badge.

One caution on the hybrid: it only stays sensible if someone owns the routing. A hybrid nobody polices decays into “whatever each team found easiest”, which is the worst of both worlds, private infrastructure carrying the cost while the sensitive workloads drift onto whichever cloud tool had the nicest interface. The routing rules need a name against them, a place in the induction material, and a periodic check that reality still matches the diagram. Ten minutes a quarter, and the architecture keeps meaning something.

What the board should actually ask

The useful board conversation isn’t cloud versus on-prem. It’s a handful of questions management should be able to answer. Which of our data can leave our control, and says who? What are we actually using AI for today, and which pile does each use touch? What does our usage cost now, and where does the curve cross the cost of owning the infrastructure? And if we had to move providers or bring a workload in-house next year, what would that take?

Add a fifth for the minutes: who verifies that the promises are being kept? Contractual guarantees about regions, retention and access are only as good as someone occasionally checking them, and “the vendor assured us” has a short shelf life in an incident review. A standing agenda line once or twice a year, evidence sighted, not assurances received, is cheap governance for a risk this shaped.

An executive team that can answer those has made the decision properly, whatever they chose. One that answers with a vendor’s slide deck hasn’t decided anything; it’s been decided for.

So don’t treat it as a side you pick for life. Decide it deliberately, with the data sitting in front of you, and write the decision down with its reasons, so that when the numbers or the obligations change, whoever revisits it can see what it was actually based on. If you want help running the sorting exercise, or a straight assessment of whether your workloads justify private infrastructure or the cloud is honestly fine, that’s a conversation we have often, and the answer we give depends on your data, not on what we’d prefer to build.