Model Extraction Over the API: Stealing a Model You Never Shipped


LLMs leak training data through query access alone — membership inference and model inversion. The defence architecture: reduce memorisation, limit the signal.
In February 2024 security researchers disclosed that a popular text-classification model on HuggingFace had been quietly modified to execute a reverse-shell payload at import time. Six months later a similar pattern hit a fork of a widely-used embedding model. Twelve months later JFrog disclosed more than a hundred malicious models sitting in plain sight on the public registry. The enterprise response in most cases was a shrug and a reminder to "be careful what you download" — which is the same response the software industry gave to npm and PyPI for a decade before SBOM-and-provenance discipline finally became table stakes. The AI supply chain is the next attack surface. This article is the deep-architecture guide: the four asset categories (weights, datasets, custom-code adapters, dep tree), the pickle problem precisely (why from_pretrained is arbitrary-code execution), the safetensors migration, CycloneDX ML-BOM, Sigstore/Cosign/in-toto for models, the Linux Foundation Model Signing project, the reference ingestion-gate architecture (picklescan + modelscan + CVE scan + provenance verify + private registry + fleet-side signature verification), dependency confusion in ML packages, embedded backdoors and stealth fine-tunes, dataset lineage and the EU AI Act, eight anti-patterns, five-stage maturity ladder, and the Monday-morning checklist.
Deepfake phishing defence for 2026: layered detection, C2PA content provenance, and the out-of-band callback protocol that defeats a flawless voice or video impersonation.
Even an aligned model under prompt injection emits unsafe output. Output guardrails are the deterministic layer that validates every response before it becomes consequential.
Advertise and partner
Reach engineering leaders and CTOs reading about AI, mobile and cloud architecture.
Sponsored mentions and co-authored articles or technical deep-dives. Relevant products only, always disclosed.