On-device AI: why useful models are getting smaller

For the last few years, “using AI” has meant sending your words to a data centre and waiting for an answer. A quieter shift is moving a growing share of that work onto the device in your hand. For a small business that handles client information, it may turn out to be the more important trend.

What changed

Two things happened at once. Chips got a new part, and models got smaller without getting useless.

• The hardware. Modern laptops and phones include a neural processing unit (NPU), a chip designed for AI work. Microsoft's “Copilot+ PC” label, introduced in 2024, requires one capable of at least 40 trillion operations per second. Apple and Google build equivalents into their own chips.
• The models. Small language models — a few billion parameters rather than hundreds — now handle everyday language tasks well. Apple Intelligence runs a model of roughly three billion parameters on the device, and in 2025 Apple opened it to app developers. Google ships Gemini Nano on Android phones for the same purpose.

Why it matters for a business

• Privacy. If the text never leaves the device, there is no third party holding it, no question about where it is stored, and far less to explain in your privacy notice. For anyone handling medical, legal, financial or HR material, that is significant.
• Cost. Once you own the hardware, running the model costs nothing per use.
• Speed. No round trip to a server. Rewriting a paragraph or summarising a document happens almost instantly.
• Offline. It works on a train, on a site visit, or in a building with poor signal.

What it is good for, and what it is not

Small on-device models are good at the frequent, modest tasks: summarising an email thread, tidying the tone of a reply, transcribing a voice note, pulling dates and names out of a document, sorting notifications. They are weaker at deep reasoning, long documents, specialist knowledge and anything needing current information from the web.

In practice, the direction is hybrid: simple requests handled on the device, harder ones sent to a larger model in the cloud. Apple's version sends those to what it calls Private Cloud Compute, designed so that the request is not stored. Knowing which of your tasks go where is worth a few minutes of reading in each tool's settings.

What to do with this

1. If you are replacing laptops anyway, a model with a capable NPU is a reasonable thing to prefer. Do not replace working machines for it. (If you are still running Windows 10, you have a more pressing reason — see Windows 10 end of support.)
2. Check where your AI tools send data, especially anything that touches client information. “On-device” and “cloud” are very different answers to a data-protection question.
3. Use on-device features for the sensitive, simple jobs — summarising a client email, transcribing a meeting note — and keep cloud tools for the heavy lifting where the data is not sensitive.

The bigger picture

The first wave of AI concentrated capability in a handful of huge data centres. This wave spreads a useful slice of it back out to ordinary devices. For small businesses, that is good news on cost, on privacy, and on independence from any single provider.

For a grounded view of where AI earns its keep and where it does not, read where AI actually helps a small business. If you want help deciding which tools are safe for your client data, ask us.

Sources

• Apple Machine Learning Research — On-device and server foundation models, 2025
• Apple Developer — Meet the Foundation Models framework (WWDC25)