On this page11 sections
- What is an on-premise AI agent?
- Why are businesses asking for on-premise AI agents now?
- What does Inkling 2 Mini change for local AI agents?
- What hardware do you need to run AI agents locally?
- How good is Inkling 2 Mini at agent work?
- What can go wrong when an agent runs on your own machine?
- How do you set up on-premise AI agents safely?
- When should you not run AI agents on your own hardware?
- What do the rules say in the EU, the US and Canada?
- Your checklist for this week
- Questions people ask
For the last two years, the honest answer to "can we run AI agents on our own servers?" was "yes, if you can afford a rack of GPUs." The open models that were good at agent work were big, and the small ones lost track halfway through a task.
That changed this week. Thinking Machines Lab released Inkling 2 Mini, a model small enough for a laptop that scores close to much larger models on agent tests. For a business that wants on-premise AI agents, where customer files, contracts and ledgers never leave the building, this is the first time the hardware question has a cheap answer.
This guide explains what an on-premise AI agent is, what Inkling 2 Mini actually changes, what hardware you need, what goes wrong, and the seven guardrails to set before an agent is allowed to act.
What is an on-premise AI agent?
An AI agent is software that uses an AI model to decide its own next step: it reads a goal, picks a tool, checks the result and repeats until the job is done. Agentic AI is the wider label for systems built that way. An on-premise AI agent runs all of this, model included, on hardware you own, so prompts, files and results stay on your network.
A few more terms you will see:
- Tool call: the agent asking a program to do something, such as "search the invoices folder" or "draft a reply."
- MCP (Model Context Protocol): in its maintainers' words, "an open-source standard for connecting AI applications to external systems." An MCP server can run on the same machine as the agent, so tools for local files or a local database work with no internet connection.
- Open-weight model: a model whose files you can download and run yourself, as opposed to one you can only reach through a vendor's API.
Why are businesses asking for on-premise AI agents now?
Agents need access to the systems that hold your most sensitive data: email, contracts, customer records, finance. That is exactly the data many firms will not send to an outside AI service.
- of 203 IT decision-makers would pick on-prem, private or hybrid over public cloud for AI on sensitive data (Cloudian, vendor-commissioned, 2026)
- 91%
- say data-residency concerns delayed or scaled back AI projects (same Cloudian survey)
- 58%
- of agentic AI projects will be cancelled by the end of 2027, Gartner predicts, citing cost, unclear value or weak risk controls
- 40%+
McKinsey's 2026 State of AI survey of 1,719 people found 40% of companies with more than $1 billion in revenue say they are scaling AI agents, against 22% of smaller firms. Agents are moving into real work, and the failures Gartner expects are mostly about cost and control, not model quality.
If you are still deciding between a fixed workflow and a free-roaming agent, read AI automation vs AI agents first. Most business processes need less autonomy than people assume.
What does Inkling 2 Mini change for local AI agents?
Three things, all from the model card and announcement:
- It fits on one machine. The model has 26.4 billion parameters in total but uses only 3.4 billion for each word it writes. The download is about 20 GB because it was trained at low precision from the start.
- It was built for agent work. Thinking Machines says it leads open models under 100 billion parameters on agentic coding. Its intended uses include agents, tool use, coding assistants and search over your own documents.
- The licence is simple. The weights are under Apache 2.0, which allows commercial use. A separate acceptable use policy is linked from the model card, so read it before you ship a product.
It also takes images as input, so an agent can read scanned invoices or screenshots, and it supports a context of up to 1 million tokens on paper. It is a "looped" model: some of its layers run three times for each word, which gives it more thinking per word without more memory for weights.
The launch demo makes the point. Thinking Machines showed all 20 residents of a simulated village, "Thinkyville", managed by Inkling 2 Mini on a single MacBook Pro with 48 GB of memory, in airplane mode. That is a demo, not a business workload, but it shows many agents running with no cloud at all.
Do not confuse it with Inkling 2 Small, a much larger 594-billion-parameter sibling released earlier in November, or with unofficial "Inkling-mini" uploads on Hugging Face.
What hardware do you need to run AI agents locally?
Memory is the limit. Here is what the model card and early testers report for Inkling 2 Mini.
| Machine | Runs Inkling 2 Mini? | Notes |
|---|---|---|
| Apple silicon Mac, 32 GB or more | Yes, official minimum | Ollama support at launch was for Macs with 32 GB or more |
| NVIDIA DGX Spark (one unit) | Yes, officially listed | A 128 GB desktop AI machine |
| PC or server GPU with 32 GB+ video memory | Yes, recommended | Serve with vLLM, SGLang or llama.cpp |
| GPU with 24 GB | A 4-bit build (about 17 GB) fits | Little room left for long tasks |
| Mac with 16 GB | Only a heavily compressed build | Expect a clear drop in quality |
| Typical office Windows laptop | Not practical yet | No large GPU, and Ollama's launch support was Mac-only |
Early users on Hacker News reported roughly 20 to 29 tokens (pieces of words) per second on a 64 GB Mac and an AMD Ryzen machine. Treat those as anecdotes, not benchmarks. One commenter also pointed out that the looped layers each keep their own working memory for the conversation, so long agent sessions need more headroom than the 20 GB download suggests.
For the wider sizing rules, server prices and break-even maths, see our guide to on-premise AI for business.
How good is Inkling 2 Mini at agent work?
On its maker's own tests, very good for its size. On independent tests, good but not frontier-class.
- on MCP Atlas, a tool-use benchmark (Thinking Machines, vendor-reported)
- 82.7%
- on the Artificial Analysis Intelligence Index, tied with Qwen3.8 27B (independent)
- 20
- per task hosted, vs $3.53 for Qwen3.8 27B (Artificial Analysis)
- $0.59
Artificial Analysis found it matched Qwen3.8 27B at its highest reasoning setting with about five times fewer active parameters, and finished tasks about four times faster. Those costs use Thinking Machines' hosted API prices, not the cost of running it on your own machine. The same test shows its limits: a closed model, GPT-6 Luna, scored 28 at about $0.15 per task.
Thinking Machines is open about the weak spots. Qwen3.8 27B beats it on office-automation tests, on instructions with many rules, and on facts: Inkling 2 Mini avoided hallucinating in 63.4% of cases against Qwen's 71.4%. The lab says plainly that looping does not add knowledge. So give a local agent your documents to search, and do not rely on what the model remembers.
What can go wrong when an agent runs on your own machine?
Running locally solves one problem: where your data goes. It does nothing for what the agent does with it. The model card itself warns about:
- Hallucinations, especially on facts.
- Long reasoning loops that burn time without finishing.
- Destructive actions. In the card's words, it "may take destructive actions, such as deleting or overwriting files, without confirmation." Thinking Machines' internal tests put the rate at 10.2%, against 35.2% for Qwen3.8 27B. Those are the vendor's own tests, with no published method.
- Weaker results outside English.
Early testers saw the same things. One Hacker News user watched an agent make more than 180 tool calls in a loop on a data-analysis task. Another saw the model spend 19,200 thinking tokens, the limit they had set, without finishing. The model card also does not document its tool-calling format yet, and older llama.cpp builds cannot load looped models at all.
The model card's own advice is short: "Run agentic workloads in a sandbox, and require confirmation before destructive or irreversible actions," and set limits on output tokens and agent turns. None of this is a reason to avoid local agents. It is the reason the harness matters more than the model.
How do you set up on-premise AI agents safely?
Here is the order we recommend for a first pilot.
- Pick one narrow, read-heavy job. Good first jobs: sorting support emails, pulling fields from invoices, answering questions from a policy folder. Avoid anything that pays, deletes or sends on day one.
- Choose the machine. A 32 GB or larger Mac, or a DGX Spark, is enough for one team's pilot.
- Serve the model behind an OpenAI-compatible endpoint. The model card shows
vllm servedoing this on port 8000, so most agent frameworks only need a new base URL. Pin a runtime version you have tested. - Connect tools with the least access possible. Start read-only. Give the email tool permission to draft, not send. OWASP's 2026 list of agent risks calls this "least agency": do not give an agent autonomy it does not need.
- Add an approval gate. Anything that deletes, pays, sends or changes records waits for a named person. OpenAI's agent guide says the same: sensitive, irreversible or high-stakes actions "should trigger human oversight."
- Set hard budgets. Cap tool calls per task, thinking tokens and run time. Anthropic calls these stopping conditions, "such as a maximum number of iterations." Pick the numbers from your own tests; the 180-call loop above is what happens without them.
- Block outbound internet and log every step. Treat any downloaded model as untrusted software, and keep a record of every prompt, tool call and result.
Then test with 20 tasks you already know the answers to, and run the same 20 through a cloud model. That tells you what you give up for privacy.
When should you not run AI agents on your own hardware?
- You need the best possible accuracy or broad knowledge. Frontier cloud models still score clearly higher.
- Cost is your only reason. Per task, a strong cloud API can be cheaper, and idle hardware still costs money.
- Many people will use it at once. One laptop serves one team's pilot, not a company.
- Nobody owns it. Someone has to patch it, watch the logs and answer when it stops.
Run agents locally when the data cannot leave, when a client or regulator demands it, or when you want to control which model runs and when it changes. Many businesses end up hybrid: sensitive work stays local, the rest goes to the cloud. Our guide to private AI for business explains the options in between.
What do the rules say in the EU, the US and Canada?
This is general information, not legal advice. Keeping data on your own hardware helps with data-residency demands, but GDPR, US state privacy laws and Canada's PIPEDA still apply to the data wherever it sits. In the EU, the AI Act's AI-literacy duty has applied since February 2025, so staff who supervise an agent need training. Whatever the region, a log of what the agent did and who approved it is the evidence you will be asked for.
Your checklist for this week
- Write down one job an agent could do with read-only access.
- Check whether you have a Mac with 32 GB or more, or budget for a DGX Spark.
- Read Inkling 2 Mini's licence and acceptable use policy.
- List every action that must wait for a person.
- Decide your limits: tool calls, tokens, run time.
- Prepare 20 test tasks with known answers.
The model that runs agents on your own hardware now fits on a laptop. The work that makes those agents safe still takes planning. If you want help choosing the first job and building the guardrails, tell us your goal or see how we work. For the costs of a full rollout, read how much AI implementation costs.
Sources
- Thinking Machines Lab, Inkling 2 Mini, November 2026
- Hugging Face, thinkingmachines/Inkling-2-Mini model card, November 2026
- Thinking Machines Lab, Inkling 2 Small, November 2026
- Artificial Analysis, Inkling 2 Mini analysis, November 2026
- Hacker News, Inkling 2 Mini discussion, November 2026 (user reports)
- eesel AI, Inkling 2 Mini overview, November 2026
- Fello AI, Running Inkling 2 Mini on a Mac, November 2026
- Neurohive, Thinking Machines opens looped model Inkling 2 Mini, November 2026
- Model Context Protocol, Introduction and Transports
- Gartner, Over 40% of agentic AI projects will be canceled by end of 2027, June 2025
- McKinsey, The state of AI, August 2026
- Cloudian, Enterprises repatriating AI workloads, March 2026 (vendor-commissioned)
- OWASP, Top 10 for Agentic Applications 2026, December 2025
- OpenAI, A practical guide to building agents, January 2026
- Anthropic, Building effective agents, December 2024
Questions people ask
Can you run AI agents on your own hardware?
Yes. Since November 2026, open models such as Thinking Machines' Inkling 2 Mini can run tool-using AI agents on one Apple silicon Mac with 32 GB of memory or one NVIDIA DGX Spark, with no data leaving your network. The model is the easy part. You still need tool permissions, a person approving risky actions, limits on tool calls and a full log.
Can AI agents work offline?
Yes, if every part is local. The model, the agent loop and the tools must all run on your machine, for example Inkling 2 Mini served on a Mac plus MCP servers for local files or a local database. Thinking Machines demonstrated 20 agents running on one MacBook in airplane mode. Any tool that calls a web service, such as web search or cloud email, still needs the internet.
What hardware do I need to run Inkling 2 Mini?
Its model card lists an Apple silicon Mac with 32 GB or more of memory, one NVIDIA DGX Spark, or another GPU with 32 GB or more of video memory. The download is about 20 GB. Compressed 4-bit builds are about 17 GB and fit some 24 GB cards, but long conversations need extra memory on top.
Is Inkling 2 Mini free for commercial use?
The weights are released under the Apache 2.0 licence, which allows commercial use, changes and redistribution. Thinking Machines also links a separate acceptable use policy, so read it before you build a product on the model. Running it yourself costs nothing in licence fees, but you pay for hardware, power and the people who look after it.
What is the difference between AI agents and agentic AI?
An AI agent is a single piece of software that uses an AI model to choose its own steps and tools to finish a task. Agentic AI is the wider label for systems built this way, including teams of agents that hand work to each other. In practice people use the two terms almost interchangeably.
Are local AI agents safe?
Local agents keep your data in-house, but they are not safe by default. Inkling 2 Mini's own model card warns of hallucinations, long reasoning loops and destructive actions taken without confirmation, and early testers reported one agent making more than 180 tool calls in a loop. Approval gates, read-only access, call budgets and logging make them safe enough for real work.
Is running an AI agent locally cheaper than using an API?
Not always. Artificial Analysis found hosted Inkling 2 Mini cost about $0.59 per task in its tests, while a stronger closed model cost about $0.15. Local hardware has no per-task fee, but it costs money up front and while idle. Run agents locally for privacy and control, and check the maths on your own workload.




