Local models and API calls: what leaves the machine

There are two places a language model can run. One is a machine you operate, reading a weights file from local disk. The other is a machine a vendor operates, reached with an API call. The prompt's privacy follows that choice, and so does the question of which model you will get next Tuesday.

The file tools on this site are a third pattern, and a smaller one. How data is processed describes PDF and image work that stays in the browser. A chat completion is not that pattern. The prompt is the product.

What an API call sends

The request body contains the messages, and often a system instruction, tool definitions, and settings such as temperature. The vendor's servers run the model and return tokens. Their privacy policy and data-processing terms are the rules for retention, training use, and who on their staff can see the text. A setting in the console may opt out of training. The text still arrived.

Logs are easy to forget. A gateway, a proxy, and an error reporter can store the prompt even when the model vendor says they do not train on it. If the document is a contract or a medical note, the list of processors includes every hop, not only the model brand on the button.

What a local run keeps

A local runner loads weights into memory and generates tokens on that hardware. The prompt does not need to cross the network for the math to happen. That is the reason people run a model for drafts of private documents, for a machine that is off the network, and for a test that has to hit the same file every time.

The first launch is often a download. Weights are large. After that, the file sits on disk until you replace it. Disconnect the network and send a prompt. If the reply still arrives, the generation itself did not need a round trip. If the app shows an error or a long pause and then a reply, it may be falling back to an API. The badge on the homepage is not the test.

Telemetry is a separate channel. A local model can generate tokens on the device while the app reports usage, account status, or the text of the prompt. Read the network section of the app, or watch a connection list once, before you treat the install as private.

Hardware, without a fake shopping list

The model card or the repository page states the file size and the precision (a common pair is 16-bit and 4-bit quantized). Memory has to hold the weights and the context you asked for. A GPU speeds generation. A machine that swaps to disk will run, slowly, or it will fail.

I am not going to publish a table of "this model needs this many gigabytes." Those numbers change with the quantization, the context length, and the runner. Use the card that shipped with the file you downloaded. If the card and the file disagree, trust the file size you can see on disk and the runner's own error when memory runs out.

Which one fits the job

Use an API when the model is only offered as a service, when the weights will not fit, or when the text is already allowed to leave. Pin the model id if you need yesterday's behavior.

Use a local file when the prompt should stay on the machine, when you need the identical weights next month, or when the network is the thing you are trying to avoid. Keep the license next to the file. Open-weight licenses are not all the same, and a download is not permission to ignore the one that came with the weights.

A hybrid is common and worth naming. The app runs a small model locally and sends "hard" prompts to an API. That is an API call for those prompts. The privacy write-up has to say so.

A check before you call a setup private

Write down four facts from the machine, not from the homepage.

The weights file: path, size, and the revision or hash printed by the runner. If you cannot find a file, you are probably calling an API.

The license file that sat next to those weights. A later section of this site covers how that license differs from an API agreement. Keep the copy you read.

The network: with the weights already on disk, disconnect and send a short prompt. A reply means generation did not need the network. A spinner that ends in a reply after you reconnect means something else was called. A crash log or an account check can still be a network call even when generation is local. One successful offline reply does not prove the app never sends the prompt. It proves this prompt was answered locally.

The fit: if the runner errors on memory, the card's size was the relevant number, and a smaller quantization or a smaller model is the next experiment. Do not "solve" a memory error by pasting the same private document into an API you did not mean to use. That swap is the privacy decision, and it belongs in the note.