Model Extraction Over the API: Stealing Your Model

Q: What is model extraction and how is it different from a data breach?

Model extraction, also called model stealing, is an attack in which an adversary queries a machine learning model's prediction API and uses the returned outputs to reconstruct the model — either a functionally equivalent copy that mimics its input-to-output behaviour, or, in sharper attacks, the actual parameters of one or more layers. Critically, no server is breached and no weights leak: the attacker uses the API exactly as a legitimate customer would, just at high volume and with queries chosen to maximise how much they learn. That is what makes it fundamentally different from a data breach. A breach is an unauthorised access event you can, in principle, detect and patch — someone got into a system they should not have. Extraction is authorised access used against you: every prediction you return is a labelled training example, so the very act of serving good answers is what leaks the value. The asset stolen is not a file but the training investment embodied in the model's behaviour. This is why perimeter security, encryption at rest and access control — the controls that stop breaches — do nothing against extraction, and why the defence has to happen at the response and traffic-analysis layer instead.

Q: How much does it cost to steal a model through its API?

Far less than people assume, and the cost scales with how much information each response leaks rather than with the size of the model. The clearest public benchmark is the 2024 work by Carlini and colleagues, "Stealing Part of a Production Language Model," which exploited the logit information exposed through a production API to recover the embedding projection matrix — the final layer — of real OpenAI models. The reported figures are striking: the full projection matrix for the Ada and Babbage models was extracted for under 20 US dollars in API queries, and the authors estimated that the same attack against GPT-3.5-turbo would cost under 2,000 US dollars. The attack also confirmed architectural details that had never been published, recovering a hidden dimension of 1024 for Ada and 2048 for Babbage. That is parameter extraction, the most surgical class. Functional extraction — training a substitute model that merely reproduces behaviour rather than recovering exact weights — is even more broadly achievable and was demonstrated as far back as the 2016 Tramèr et al. work on prediction APIs. The general rule is that richer outputs, such as full probability vectors, logprobs or top-k tokens, dramatically lower the number of queries and therefore the dollar cost required.

Q: Can I prevent model extraction completely?

No, and any vendor claiming otherwise is selling reassurance rather than a control. The impossibility is structural: the property that makes an API valuable — returning good answers to queries — is exactly the property an attacker harvests to clone the model. A model that answered badly enough to be un-extractable would be useless to your paying customers too. So the realistic goal is not prevention but attrition: raising the number of queries, the detection risk, and the legal exposure of an extraction campaign until it becomes cheaper for a rival to train their own model from scratch than to steal yours. You do that with layered defences — minimising the information each response returns, imposing per-principal rate limits and spend caps so bulk querying hits an economic wall, watermarking outputs so a distilled model carries provable evidence of theft, and running behavioural analytics to detect the coverage-and-volume signature of a campaign and degrade fidelity for suspected extractors. Each layer is a cost multiplier, not a barrier. The correct mental model, familiar from the rest of security, is that you are pricing the attack out of viability rather than locking a door, because in the API setting there is no door to lock.

Q: What is the difference between model extraction and knowledge distillation?

They are the same mechanism applied with opposite legitimacy. Knowledge distillation is a well-established, legitimate technique in which you train a smaller student model to mimic a larger teacher model you own or are licensed to use, transferring the teacher's behaviour into a cheaper, faster model — a standard efficiency move for reducing inference cost and latency. Model extraction is that identical teacher-student process turned against a model you do not own: the attacker treats someone else's API as the teacher, harvests its outputs as training labels, and distils a competing model without permission and without paying for the training the victim invested in. Technically the pipelines look alike — query a model, collect input-output pairs, train on them — which is precisely why extraction is so hard to distinguish from heavy legitimate use, and why the recent commercial disputes over whether one lab distilled another's model through its API are genuinely difficult to adjudicate on traffic alone. The distinction is ownership and consent, not method. This is also why watermarking matters: because the mechanism is indistinguishable, the durable defence is to mark your outputs so that a model trained on them can be shown, statistically, to have consumed them.

Q: Why does returning logprobs or probabilities make extraction easier?

Because every additional bit of information in a response is a bit of gradient signal an attacker can use, and confidence outputs are extremely information-dense. When an API returns only a final label or a single generated token, the attacker learns one discrete fact per query. When it returns full class probabilities, logprobs, or top-k token distributions, each query reveals not just the answer but how confident the model is and how it ranks the alternatives — which exposes the shape of the model's decision surface, not just a point on it. That richer signal collapses the number of queries needed to reconstruct behaviour, and in the parameter-extraction case it is the exact ingredient that makes recovering a layer's weights tractable: the Carlini attack depended on the logit information the API exposed. This is why the single highest-leverage defensive move is at the output layer — return the least information that still serves the genuine use case. Dropping logprobs and top-k logits from public tiers, removing logit_bias, and rounding or adding calibrated noise to any confidence score you must expose removes the cheap parameter-extraction attacks outright. It does not stop an attacker distilling your plain-text outputs, but it raises the cost of the most surgical attacks from trivial to substantial.

Q: Does watermarking stop model extraction?

No — watermarking does not prevent extraction, it makes extraction provable after the fact, which is a different and complementary job. A statistical text watermark, such as the green-list token-biasing scheme introduced by Kirchenbauer and colleagues in 2023, subtly steers the model's token selection into a pattern that is invisible to a reader but detectable by a statistical test if you know the key. The value is attribution: if a competitor distils a model on your watermarked outputs, the resulting model tends to inherit the watermark signal, so you can later test a suspect model and produce statistical evidence that it was trained on your responses. That converts a Terms of Service clause from an unenforceable wish into a claim you can actually support, which is why watermarking is the layer that gives your legal position teeth. But on its own it stops zero queries — the attacker still extracts freely — and watermarks can be weakened by aggressive paraphrasing or heavy fine-tuning of the stolen model. So watermarking belongs in the stack as the attribution layer, sitting alongside response minimisation, rate and spend limits, and campaign detection. Treating it as a standalone defence is an anti-pattern; treating it as the thing that makes your other remedies enforceable is correct.

Q: How do I detect a model-extraction campaign in my API traffic?

You detect it by looking for the emergent shape of a campaign across many requests, because no single query is distinguishable from legitimate use. The signals are statistical and behavioural rather than content-based. Extraction traffic tends to cover the input space unusually systematically, sweeping across regions of possible inputs to map the model's behaviour comprehensively rather than following the clustered, task-driven patterns of real users. It often probes decision boundaries, sending near-duplicate queries that differ only slightly to find exactly where the output flips — a strong signature of someone mapping the model rather than using it. And it sustains high volume per principal over time. Practical detection therefore aggregates per API key or per authenticated identity: query entropy and diversity, coverage of the input distribution, the rate of boundary-probing near-duplicates, and total volume against a baseline of normal use. When those cross a threshold, the right response is graduated rather than binary — issue a challenge, degrade output fidelity, throttle, or ultimately block — so you avoid punishing a legitimate heavy user on a false positive. This is fundamentally an analytics and identity problem, which is why per-principal authentication and a governed gateway are prerequisites: you cannot profile traffic you cannot attribute to a stable identity.

Q: Is a Terms of Service clause enough to protect against model theft?

A Terms of Service clause prohibiting distillation or extraction is necessary but nowhere near sufficient, and relying on it alone is one of the most common anti-patterns. A ToS is a legal instrument, not a technical control: it stops no queries, detects no campaigns, and only becomes useful after an extraction has already happened, in a dispute or in court. Worse, it is only enforceable if you can actually prove that a given model was trained on your outputs, and proving that is hard precisely because extraction is mechanically identical to legitimate use — which is exactly the gap that output watermarking exists to close. Without watermarking or comparable provenance evidence, a distillation clause is, in effect, a wish with a lawyer attached. The defensible posture treats the ToS as the final layer of a stack whose technical layers do the real work: minimise the information each response leaks, impose rate limits and spend caps that make bulk extraction economically painful, watermark outputs so theft leaves detectable evidence, and detect campaign-shaped traffic to degrade it in real time. In that context the ToS has teeth, because you can meet its evidentiary burden. On its own, it is documentation of your intentions, not protection.

Satyam Kumar

Retour au blog

cyber-security-patterns

Model Extraction Over the API: Stealing a Model You Never Shipped

By Satyam KumarAugust 21, 202610 min read

model extraction model stealing ai security llm security inference api security model theft knowledge distillation llm watermarking machine learning security ml model protection api rate limiting adversarial machine learning owasp ml top 10 ai gateway model provenance 2026

Model Extraction Over the API: Stealing a Model You Never Shipped

Frequently Asked Questions

Partager l'article

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

Architecte AI & Cloud. J'aide les équipes à construire des systèmes qui scalent pour des millions.

LinkedIn GitHub

Model Extraction Over the API: Stealing a Model You Never Shipped

Frequently Asked Questions

Partager l'article

Comments

Leave a comment