Third-Party AI Model Due Diligence: The 3-Ledger Method

Q: What is third-party AI model due diligence?

It is the process of deciding whether an externally provided model, usually a hosted API, is fit for a specific use in your organisation. It should produce three kinds of evidence: verified vendor claims about security and data use, behaviour you measure yourself on your own task, and contract terms that govern how the model can change after you sign.

Q: Why is a vendor security questionnaire not enough for an AI model?

A questionnaire records claims about the vendor: certifications, policies, data handling. It says nothing about how the model behaves on your documents, how often it invents facts in your domain, or what happens when the vendor swaps weights behind the same endpoint. Those risks need a scored evaluation on your own data and contract clauses on change, neither of which a questionnaire can provide.

Q: When did EU AI Act obligations for general-purpose AI models start?

According to the European Commission AI Act page, obligations for providers of general-purpose AI models have applied since 2 August 2025. The Commission published the General-Purpose AI Code of Practice on 10 July 2025, with chapters on transparency, copyright, and safety and security, the last applying only to models with systemic risk. Downstream buyers can reasonably expect substantive model documentation.

Q: Did the EU AI Act high-risk deadline move from August 2026?

Yes. The European Commission page now states that, following the AI Omnibus simplification agreement, high-risk rules for Annex III use cases apply from 2 December 2027, and for AI systems embedded in regulated products under Annex I from 2 August 2028. The general-purpose AI obligations did not move. Earlier articles that framed August 2026 as the high-risk deadline are out of date.

Q: How big should an evaluation set be for vendor model due diligence?

Two hundred to five hundred cases drawn from your real task is usually enough to separate candidate models on the failures you care about. Include real documents with sensitive fields removed, domain edge cases, and adversarial prompts. Score each case pass or fail against a written rubric. Small sets miss rare failures, so pair them with production sampling and human review.

Q: Which contract clauses matter most when buying an AI model API?

Four matter most: a written exclusion of your inputs and outputs from training, a stated retention period for abuse monitoring with zero-retention options, a minimum deprecation notice period, and the ability to pin a dated model version rather than a floating alias. Add a commitment to notify material behaviour changes and your right to re-test. Without these, approval expires at the next vendor deploy.

Q: How should AI model use cases be tiered for due diligence?

Tier by consequence. Tier 1 is internal work on non-confidential material and needs a claim review plus gateway logging. Tier 2 involves personal or confidential data or customer-facing output and needs a scored evaluation and data processing addendum review. Tier 3 covers decisions about people or regulated products and needs all three ledgers, negotiated change terms and re-testing on every version change.

Q: When should you refuse to approve a third-party AI model?

Refuse it for confidential, customer-facing or high-risk use if the vendor will not commit in writing to excluding your data from training, will not give a deprecation notice period, or will not let you pin a model version. A strong demo does not compensate for an unbounded change ledger. Such a model can still be acceptable for low-risk tier 1 internal use, or you walk away.

Satyam Kumar

Retour au blog

architecture

The Questionnaire Is Not the Evidence: Third-Party AI Model Due Diligence in 2026

By Satyam KumarOctober 28, 202612 min read

ai vendor due diligence third-party ai risk ai model procurement ai vendor assessment eu ai act gpai gpai code of practice ai governance model risk management llm evaluation ai vendor questionnaire data processing addendum ai model version pinning ai gateway nist ai rmf ai risk tiering enterprise ai procurement

The Questionnaire Is Not the Evidence: Third-Party AI Model Due Diligence in 2026

Frequently Asked Questions

Partager l'article

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

Architecte AI & Cloud. J'aide les équipes à construire des systèmes qui scalent pour des millions.

LinkedIn GitHub

The Questionnaire Is Not the Evidence: Third-Party AI Model Due Diligence in 2026

Frequently Asked Questions

Partager l'article

Comments

Leave a comment