Skip to content

Private AI16 min read

On-premise AI for business: hardware, costs and when it is worth it

On-premise AI for business: what it is, the open models and hardware to run a private LLM, November 2026 prices, break-even maths and when the cloud wins.

Illustration: a server rack inside a building outline with a glowing AI chip above it, cabled to three team members.
On this page13 sections
  1. What does on-premise AI mean?
  2. Where should your AI run? Three questions
  3. Which open models can a business run on its own hardware?
  4. How much memory does a model need?
  5. What does on-premise AI hardware cost in November 2026?
  6. Which software runs the model?
  7. Is on-premise AI cheaper than the cloud?
  8. How many companies are moving AI in-house?
  9. What are the costs people forget?
  10. What about private cloud and hosted options instead?
  11. What do the rules say in the EU, the US and Canada?
  12. How do you test in 30 days before you buy?
  13. Questions people ask

"We want our own AI, on our own hardware" is a common request, and sometimes it is the right call. This guide is built like a shopping and decision pack instead of an essay: what on-premise means, which open models you can run, three example builds with current prices, the break-even maths, the costs people forget, and a 30-day plan that lets you test before you spend.

One warning first. Hardware prices are moving fast. Memory shortages pushed several prices up sharply through 2026, so treat every figure here as a November 2026 snapshot and re-check before buying.

What does on-premise AI mean?

On-premise AI (also written on-premises) runs the model on servers or workstations that you own, in your building, a colocation rack or your own data centre. Nothing goes to an outside AI vendor. It is one point on the private AI scale. Our guide to what private AI is covers the full range, from business plans to private cloud to on-premise.

Three terms people mix up:

  • Private LLM: a language model you control. It can run on-premise or in your own cloud account.
  • Private GPT: an informal name for a ChatGPT-style assistant that answers from your own documents.
  • Self-hosted or local LLM: an open-weight model you run yourself, whether on a laptop, a workstation or a server.

Where should your AI run? Three questions

Where should your AI runThree questions decide where private AI should run. If data must stay on site, run it on-premises or air-gapped. If not, and your daily use is heavy and steady, run your own GPU server on-site or colocated. If not, and data must stay in your country, use a private cloud in your region. Otherwise a business-tier SaaS plan with a signed data processing agreement is enough.Must data stay on site?yesOn-premisesor air-gappednoHeavy, steady daily use?yesOwn GPU serveron-site or colocatednoMust data stay in-country?yesPrivate cloudin your regionnoBusiness-tier SaaSwith a signed DPA
Start with the legal and contractual answer, then the usage pattern. Hardware comes last.

The order matters: law and contract first, usage second, hardware last. If nothing forces data onto your premises, a private cloud in your region is almost always faster and cheaper to start.

Which open models can a business run on its own hardware?

Open-weight models changed fast through 2026. These are the main families with licences that allow commercial use, checked on the vendors' own pages in November 2026. Always read the licence for your own case.

Model familySizes (total / active)LicenceNotes
OpenAI gpt-oss (Aug 2025)117B / 5.1B and 21B / 3.6BApache 2.0, plus usage policyOpenAI says the larger fits in 80 GB and the smaller in 16 GB
Alibaba Qwen3.8 (Aug 2026)27B dense, plus a much larger flagshipApache 2.0 for the 27B262K-token context. The flagship has a custom licence we could not read
Google Gemma 4 (Apr 2026)E2B, E4B, 12B, 26B, 31BApache 2.0The first Gemma under Apache. Small enough for a single machine
Mistral Small 4 (Mar 2026)119B / 6BApache 2.0"Small" in name only: Mistral lists 4 H100 GPUs as the minimum
Mistral Large 4 (Oct 2026)about 1T / 52BModified MITCommercial use allowed. Running it as a service for others needs a separate licence
DeepSeek V4 (Apr 2026)up to 1.6T / 49BMITVery large models need multi-GPU servers
NVIDIA Nemotron 3 (2026)120B / 12B and 550B / 55BTwo different licencesCheck which one applies to the model you pick
Meta Llama 4 (2025)Scout, MaverickMeta community licenceStill downloadable, but Meta's newest models are proprietary or have limited terms. The multimodal rights are not granted to EU-based companies
Thinking Machines Inkling 2 Mini (Nov 2026)26.4B / 3.4BApache 2.0, plus acceptable use policyAbout a 20 GB download. Runs on an Apple silicon Mac with 32 GB or one DGX Spark, and is built for agent work. See running AI agents on your own hardware

"Total / active" matters for mixture-of-experts models: only the active part does the work for each word, but all the parameters must be loaded into memory.

One security note. In October 2026 Cohere, a competitor of the model's maker, reported rare unwanted outbound network behaviour in a preview of a Qwen model during sandbox testing. Whatever you choose, treat any model as untrusted software: block outbound internet from the machine that runs it by default.

How much memory does a model need?

Memory decides what you can run. These rules come from NVIDIA's and Google's own published figures.

RuleWhat it means
WeightsAbout 2 GB per billion parameters at 16-bit, 1 GB at 8-bit, 0.5 GB at 4-bit (NVIDIA's worked example is 7B at 16-bit equals 14 GB)
Real exampleGoogle's Gemma 4 31B figures, which include about 20% overhead: 69.9 GB at 16-bit, 34.9 GB at 8-bit, 17.5 GB at 4-bit
Mixture-of-expertsLoad everything: Gemma 4 26B-A4B needs 57.7 GB at 16-bit even though only 4B parameters are active
ContextLong conversations need extra memory. NVIDIA says one user with a 128K-token context on a 70B-class model uses about 40 GB

So a 4-bit 30B-class model fits in a laptop-class 24 GB, a 120B model needs about 80 GB, and anything near a trillion parameters needs a multi-GPU server.

What does on-premise AI hardware cost in November 2026?

Here are three example builds. Prices are US dollars from vendor pages, news reports and resellers, as seen in November 2026. They are volatile, and CAD and EUR prices were not consistently available, so we have not converted them.

BuildExample hardwareListed priceWhat it runs
Starter: one person or a small pilotApple Mac Studio, M5 Max, up to 128 GBFrom $2,499. The 128 GB option adds $2,000Models up to about 100B parameters at 4-bit
NVIDIA DGX Spark, 128 GB$6,950 on NVIDIA's store, up from $3,999 at launch, shown out of stockSame class. A 64 GB version is $4,999 via partners
Framework Desktop, 128 GB$3,449, up from $1,999 at launchSame class. A 192 GB version is $6,799
Team: a department sharing one GPUNVIDIA RTX PRO 6000 Blackwell, 96 GB$16,000 for the card on NVIDIA's US marketplace in August 2026, up from about $8,500 at launchgpt-oss-120b and similar. You also need the rest of the machine
Group: several departmentsFour RTX PRO 6000 in one server, 384 GB totalAbout $117,000 to $152,000 depending on configuration (resellers, October to November 2026)Large models, many concurrent users
Frontier-class8-GPU H200 or B200 serverRoughly $340,000 to $470,000 (indicative, no NVIDIA list price)The very largest open models

Some notes on the table:

  • The Apple Mac Studio also comes in an M5 Ultra from $5,499 with up to 512 GB. The 512 GB configuration is $22,299, according to published reports.
  • The consumer RTX 5090 has a $1,999 list price and 32 GB, but street prices in November 2026 started above $6,000 at Newegg, about three times list.
  • European buyers: one reseller lists 128 GB GB10-class systems at about €5,000 to €8,100 before VAT.
  • Memory-driven price rises are the reason. Framework itself said it expects memory prices to keep rising for months.

How many people can one machine serve?

A published benchmark by an independent site (Millstone AI, January 2026) ran gpt-oss-120b on one RTX PRO 6000 with vLLM. A single user got about 181 tokens per second. With about five users chatting at once at a 32K-token context, each still got a first word in under 8 seconds and over 15 tokens per second. Peak power draw was 530 W.

The lesson: a single card handles single-digit simultaneous chats at long context. Because most staff are idle most of the time, it can serve a larger team than that, but test with your own usage. A four-card server measured about 3,956 tokens per second in total on gpt-oss-120b, according to StorageReview as cited by a reseller.

Which software runs the model?

ToolWhat it isBest for
vLLMProduction serving library with continuous batching and an OpenAI-compatible APIMany simultaneous users on GPUs
llama.cppLightweight engine with a built-in server, runs on Mac, CPU and mixed hardwareOne machine and small teams
OllamaEasiest way to run open models locally, with a REST APISingle users and small teams. Default concurrency is low
LM StudioDesktop app with a headless server modeEvaluation and small internal use. Check its current terms

To build a "private GPT" on top, you add a chat interface, retrieval over your documents, sign-in with permissions and logging. Our post on whether you can train AI on your company data explains why retrieval, not training, is usually the right approach.

Is on-premise AI cheaper than the cloud?

Usually not at low volume. Here is the arithmetic, using list prices.

When running your own AI can pay offAn illustrative chart, not real prices. Monthly cost on the vertical axis, AI usage on the horizontal axis. Pay-per-token cloud AI starts low and rises steadily with usage. Your own hardware starts higher and stays almost flat. The lines cross at a break-even point: below it the cloud is cheaper, above it your own hardware can be cheaper if the usage is heavy and steady.Monthly costIllustrative, not real pricesPay per tokenOwn hardwareBreak-evenLight useHeavy, steady use
Illustrative shape only. Your break-even depends on model size, hardware price and how steady the demand is.

A hosting company, MassiveGrid, lists a dedicated server with one 96 GB RTX PRO 6000 at $1,656.15 per month. Anthropic's Sonnet 5.5 is priced at $2 per million input tokens and $10 per million output tokens. If we assume three input tokens for every output token, the blended price is $4 per million tokens. Then:

  • $1,656 per month divided by $4 per million = about 414 million tokens a month, or roughly 19 million per working day, to break even against a mid-tier API.
  • Against a small-model API such as Claude Haiku 5.5 ($0.10 and $0.50, blended $0.20), the same server needs about 8.3 billion tokens a month.

That is a vendor's server price, a rented machine and no staff. If you buy a $16,000 card instead, it is about $444 a month over 36 months before the rest of the server, power and people. Power alone: a card drawing 600 W around the clock uses about 5,256 kWh a year, which is about €950 at the EU average non-household price of €0.18 per kilowatt-hour in the first half of 2026.

An academic analysis (Patil, arXiv, June 2026) found the same H100 hardware could cost anywhere from $0.21 to $15.25 per million output tokens depending on how busy it is, with an under-use penalty of 2.5 to 24 times at one to ten requests per second. Utilisation is the whole game.

Vendor studies from Dell claim on-premise inference is far cheaper than cloud. They are vendor-commissioned and we could not read the methodology, so we have not used their numbers.

When on-premise does not pay off

  • Light or bursty use: an API is free when idle, and an idle GPU is not.
  • You need top-tier quality: the best open models are huge and need multi-GPU nodes.
  • A weaker model needs retries, which erases the savings.
  • Nobody is costed for operations: patching, monitoring, backups, on-call.
  • Traffic spikes: an API absorbs a tenfold spike, a single card queues.
  • Hardware inflation: some prices rose 75% to 87% since launch, which lengthens payback.
  • API prices keep falling. Gartner predicts that by 2030 inference on a one-trillion-parameter model will cost providers more than 90% less than in 2025.

When on-premise makes sense

  • Law, contract or client demands keep data on site.
  • Use is heavy and steady, so the hardware stays busy.
  • You want to swap models without changing your platform.
  • You can name the person who will run it.

How many companies are moving AI in-house?

of 1,500 enterprise architects moved AI workloads from public cloud to private or on-premises in a year (Cloudera and Wakefield, vendor-commissioned)
66%
name on-premises as their main place for AI inference; hybrid 31% and public cloud 30% still lead (same survey)
8%
have already moved AI or machine-learning workloads out of public cloud (Foundry 2026 Cloud Computing Study)
34%

The Foundry study gives the reasons: security 60%, cost 42%, performance 42% and sovereignty 40%. It also found only 13% plan to run AI primarily in public cloud. Read both surveys as direction, not precision. Cloudera's own data shows most firms are choosing hybrid rather than going fully on-premise.

What are the costs people forget?

  • People: someone has to patch, monitor and fix it. This is usually the biggest cost.
  • Redundancy: one machine is one point of failure. Plan a spare or a cloud fallback.
  • Power and cooling: 600 W cards are heaters. A server rack needs a proper room.
  • Backups and logging: you now own audit trails and retention.
  • Security work: access control, network rules, patching and supply-chain checks on the models you download.
  • Software and integration: chat interface, retrieval, permissions, connectors to your documents.
  • Replacement: hardware refreshes arrive every few years, and the models you want keep growing.

IBM's 2026 breach study found that 92% of organizations that suffered an AI-related breach lacked proper AI access controls. Keeping data on-premise does not fix that. Access controls and logging do.

What about private cloud and hosted options instead?

If you want control without owning machines, there are middle options. These are summaries from the providers' pages in November 2026.

OptionWhat it offers
AWS Bedrock, European Sovereign CloudGemma 4 models, pay per token, zero operator access, zero data retention by default, inference kept in the region (announced September 2026)
Google CloudPay-per-token open models, or dedicated endpoints with your own weights
OVHcloud AI EndpointsServerless, OpenAI-compatible models hosted in Europe. Its docs say data is not stored or shared during or after model use
Cohere (Canada)Private deployment on-premise or in an isolated cloud, dedicated managed inference, hybrid and SaaS options
TELUS Sovereign AI FactoryCanadian-owned infrastructure in Quebec, reported sold out, expanding to British Columbia sites
Bell AI FabricA sovereign GPU facility in Merritt, British Columbia, announced March 2026. No GPU count was published

A sensible path: test with an open model in one of these hosted settings first, then buy hardware only if your data and usage justify it.

What do the rules say in the EU, the US and Canada?

This is general information, not legal advice. For the full picture, see what private AI means for business.

  • Keeping data on site does not remove your legal duties. GDPR, HIPAA and Canadian privacy laws apply to the data wherever it sits. On-premise helps with data-residency demands, but you become responsible for security.
  • Localisation pressure is real. Cisco's 2026 privacy benchmark (5,200 professionals in 12 markets, Canada not included) found 81% face heightened data-localisation demands, and 85% say it adds cost, complexity and risk.
  • EU: the AI Act's AI-literacy duty has applied since February 2025, with national supervision from August 2026. High-risk obligations were delayed to December 2027 and August 2028. Check the licence of any model you use: Meta's Llama 4 multimodal rights are not granted to EU-based companies.
  • US: state AI laws are the main action. Colorado's replacement law, SB 26-189, takes effect on 1 January 2027, and California's rules on automated decision-making apply from January 2026 with later compliance dates.
  • Canada: PIPEDA, the Privacy Commissioner's generative-AI principles and Quebec's Law 25 apply. Canada has no federal AI statute in force, and Bill C-36 has only passed first reading.

How do you test in 30 days before you buy?

  1. Week 1: choose one use case and one document set. Write the policy for what data is allowed.
  2. Week 1: write 20 test questions you already know the answers to.
  3. Week 2: test two or three open models in a hosted setting (private cloud or a rented GPU). Score accuracy and speed on your questions.
  4. Week 3: pilot on small hardware such as a 128 GB machine with three to five users. Measure speed, quality and how many people can use it at once.
  5. Week 4: decide. Buy, stay hybrid or stay on a business-tier plan, based on measured use, not hopes.

For the broader cost picture, see how much AI implementation costs. If you want help choosing and testing, read how to choose AI consulting services, tell us your goal or see how we work. Once the model is running, AI automation vs AI agents explains what to build on it, and can you run AI agents on your own hardware? shows how to keep a local agent safe.

Sources

Questions people ask

What is on-premise AI?

On-premise AI means running AI models on hardware you own and control, in your office or your own data centre, so prompts and documents never leave your network. It is the most controlled form of private AI. You are responsible for the hardware, security, updates and uptime, which is the main cost beyond the equipment itself.

How much does on-premise AI cost?

It depends on the model size and how many people use it. In November 2026 a single-machine pilot with 128 GB of memory ran from about $2,500 to $7,000, a 96 GB workstation GPU listed around $16,000 for the card alone, and a four-GPU server was listed at about $117,000 to $152,000. Power, staff time and backup are extra, and prices are rising.

What hardware do I need to run a private LLM?

Memory is the limit. A model needs about 2 GB per billion parameters at 16-bit, 1 GB at 8-bit and 0.5 GB at 4-bit, plus space for the conversation context. A 120-billion-parameter model such as gpt-oss-120b needs about 80 GB. Small teams can start on a 128 GB unified-memory machine. Many simultaneous users need a data-centre style GPU.

Is on-premise AI cheaper than the cloud?

Usually not for light use. A hosted 96 GB GPU server at about $1,656 per month breaks even against mid-tier API prices at roughly 414 million tokens a month, and against small-model prices only at roughly 8 billion. Self-hosting wins when use is heavy and steady, or when control matters more than cost. Idle hardware still costs money.

What is the best open-source LLM for business?

There is no single best. In November 2026 the main commercially usable families include OpenAI's gpt-oss, Alibaba's Qwen3.8, Google's Gemma 4 and Mistral's models under Apache 2.0, DeepSeek V4 under MIT, and NVIDIA's Nemotron 3. Choose by memory fit, licence terms and accuracy on your own documents, and test two or three before you buy hardware.

Is on-premise AI more secure than cloud AI?

It gives you more control, not automatic security. Data stays on your network, which helps with data-residency rules, but you must patch systems, control access, log activity and block unneeded outbound connections. IBM's 2026 breach study found 92% of organizations with an AI-related breach lacked proper AI access controls, so the controls matter more than the location.

What is the difference between on-premise AI and private cloud AI?

On-premise AI runs on hardware you own. Private cloud AI runs in your own cloud account or a dedicated environment from a provider, so you rent the machines but control the settings and keys. Private cloud is quicker to start and scales up and down. On-premise gives the most control and needs the most upkeep.

Can I build a private GPT for my company?

Yes. You need an open or dedicated model, serving software such as vLLM, llama.cpp or Ollama, a chat interface, retrieval over your documents, sign-in with permissions, and logging. Start with one team and one document set, test with 20 questions you already know the answers to, and only then scale up.

Keep reading

All posts

Want this working in your business?

Tell us what you want to get done. We will tell you if AI can help.