The draft Annex 22 is six pages, ten numbered sections and a glossary, published for consultation on 7 July 2025 with comments closing 7 October 2025, and as of August 2026 the EudraLex Volume 4 index still does not list it. Its scope narrows in four steps: computerised systems in manufacturing where AI is used in critical applications, then machine learning models trained rather than explicitly programmed, then static models only, then deterministic models only. The fifth paragraph is the consequence of the first four: generative AI and large language models fall outside the document and should not be used in critical GMP applications, while non-critical use stays permitted with a qualified human responsible for output suitability. The word should appears 62 times in the draft, must and shall appear zero times, so any comparison table rendering these clauses as mandatory is the summariser talking rather than the annex. A language model fails the determinism test before anyone reaches the generative-AI sentence, and so does any classifier served on an endpoint whose weights and batch shape you do not control. The clause that will actually fail an audit is Section 6 on test data independency: no copies of test data outside a repository with access control and audit trail, plus staff separation between whoever saw the test set and whoever trained the model. Start by inventorying every model that touches a GMP decision and sorting it with the three-impact test, because that list is needed under any final wording.
The sentence that decides this is one line in a six-page consultation draft, and almost nothing ranking for the question prints it: "Following the above, the document does not apply to Generative AI and Large Language Models (LLM), and such models should not be used in critical GMP applications." That is the draft EU GMP Annex 22 on large language models. The verb is should, the document is a draft whose comment period closed in October 2025, and the exclusion attaches to critical applications only.
It is a consequence, not a rule. It opens with "Following the above", and the above is a chain of scope filters. Anything probabilistic is out at the fourth paragraph, transformer or not. Whether the system is also regulated under the EU AI Act is a different question from GMP, which the draft never mentions.
What the draft actually says about LLMs
The draft runs six pages and holds ten numbered sections plus a glossary; its header line reads "Reasons for changes: Not applicable (new annex)." Scope paragraph 2 covers machine learning models that obtained their functionality through training rather than explicit programming. Three more paragraphs set the boundary before the LLM sentence arrives.
The LLM paragraph continues past the sentence everyone quotes, and its second half is the permission: "If used in non-critical GMP applications, which do not have direct impact on patient safety, product quality or data integrity, personnel with adequate qualification and training should always be responsible for ensuring that the outputs from such models are suitable for the intended use, i.e. a human-in-the-loop (HITL) and the principles described in this document may be considered where applicable."
The exclusion is bookkeeping: the same scope filters catch systems nobody calls generative AI.
| Model type | Critical application | Non-critical application | Deciding text |
|---|---|---|---|
| Static ML, deterministic output | In scope, applies in full | Principles optional | Scope paras 2 to 4 |
| Dynamic, learns during use | Should not be used | Not addressed | Scope para 3 |
| Probabilistic output, any architecture | Should not be used | Not addressed | Scope para 4 |
| Generative AI and LLMs | Should not be used | Permitted, qualified human responsible | Scope para 5 |
Is Annex 22 in force yet, and what a draft can do to you
The consultation opened on 7 July 2025 and closed on 7 October 2025 at 23:59 CEST, publishing a revised Chapter 4 on Documentation, a revised Annex 11 on Computerised Systems, and the new Annex 22. Status on the Commission's page: Closed.
Check the primary index, not a summary. As of August 2026 the EudraLex Volume 4 page lists 17 annexes, numbered up to 21, with no Annex 22 entry and Annex 11 still shown as the January 2011 revision. The index carries adoption dates where they exist: the revised Annex 19 is marked applicable from 24 September 2026. Annex 22 carries nothing of the kind, so treat any adoption timetable you are shown as unsourced.
The modal verbs settle the rest. Across the draft's six pages, "should" occurs 62 times, "must" and "shall" zero times. Any comparison table rendering an Annex 22 row as mandatory, the explainability row included, is the summariser talking, and clause 8.1 is itself qualified with "Where applicable".
How to decide whether your application is critical
One test, from the first scope sentence: direct impact on patient safety, product quality or data integrity. Three impacts joined by or, so one hit is enough.
Clause 3.1 says what the classification costs. The intended use description needs "a comprehensive characterisation of the data the model is intended to use as input and all common and rare variations; i.e. the input sample space", plus "Any limitations and possible erroneous and biased inputs", with a process SME responsible for its adequacy, approved "before the start of acceptance testing." Clause 3.2 splits that space into subgroups by decision output, site or equipment, material characteristics, and defect type.
Three branches cover the decision. This is our phrasing of the scope paragraphs, not a rule any regulator publishes.
# Authored template, derived from draft Annex 22 Section 1.
critical = impacts(patient_safety) or impacts(product_quality) or impacts(data_integrity)
if critical and (is_dynamic or is_probabilistic or is_generative):
disposition = "excluded, draft Annex 22 scope paras 3 to 5"
elif critical:
disposition = "in scope, full annex evidence set applies"
else:
disposition = "permitted, qualified human responsible for output"The trap is the SOP row: a drafted document signed by a human is non-critical only while the human reads it. Computer vision defect detection on the line inherits the whole evidence set; predictive maintenance does not, unless its output feeds a product-quality decision.
| Application | Impact on | Class | Evidence that follows |
|---|---|---|---|
| Visual inspection accept or reject | Product quality, patient safety | Critical | Sections 3 to 10 in full, static model only |
| Parameter prediction driving control | Product quality | Critical | Same, plus 10.1 over physical inputs |
| Deviation triage closing records | Data integrity | Critical as automated closure | Non-critical if a reviewer decides each case |
| Draft SOPs, translations, meeting notes | None once a qualified person owns them | Non-critical | Named operator, 10.5 records |
| Generative model making test labels | The qualification evidence | Not recommended, 5.6 | Use fully justified |
The determinism test an LLM cannot pass
Scope paragraph four is a claim about your serving stack, not about model architecture. Identical inputs, identical outputs.
A language model fails it three ways and only the first is sampling. Temperature 0 removes the sampler, not the variance: identical inputs still do not give identical outputs at temperature 0, because batching changes the floating-point arithmetic under a request.
The third failure mode has nothing to do with decoding and should worry a validation lead more. The glossary defines Static as "Frozen model: A model where all parameters have been finally set, not allowing further adaption to new data." A provider-managed endpoint gives you a model id, not evidence that the parameters behind it were identical for the batch you qualified and the batch you released last week. You cannot hash it or diff it, and any change lands on the provider's calendar.
Passing it would take weights you hold and can hash, a pinned runtime, a fixed batch shape, recorded seeds, and a record showing identical outputs across reruns. Achievable for a small classifier on your own hardware, not behind a hosted chat endpoint, and even then a generative model still fails on scope. What passes for critical applications is a classifier or detector, and the argument that would change that is about reproducibility, not capability.
Where LLMs are still allowed and what the draft asks in return
Non-critical use is permitted, with three clauses most summaries omit.
Clause 3.3 puts the reviewer inside the qualification record: "Where a model is used to give an input to a decision made by a human operator (human-in-the-loop), and where the effort to test such model has been diminished, the description of the intended use should include the responsibility of the operator. In this case, the training and consistent performance of the operator should be monitored like any other manual process." Reduce testing because a human checks the output and you have traded model evidence for operator evidence.
Clause 10.5 asks for records from the human review process, and says that depending on criticality this "may imply a consistent review and/or test of every output from the model, according to a procedure." Clause 5.6 closes the obvious shortcut: "Generation of test data or labels, e.g. by means of generative AI, is not recommended and any use hereof should be fully justified."
Designing the approval point is covered elsewhere. What the annex adds is narrow: the operator is named in the intended use, their consistency is monitored, and the review leaves a record.
Test data independency is the clause that will fail your supplier
Section 6 turns a hosted model from a procurement question into an evidence question. Clause 6.1 wants controls ensuring test data "is not used during development, training or validation of the model". Clause 6.2 covers the split case: where test data is split from a pool before training starts, "it is essential that employees involved in the development and training of the model have never had access to the test data. The test data should be protected by access control and audit trail functionality logging accesses and changes to these. There should be no copies of test data outside this repository." Clause 6.5 asks for staff separation or, where that is impossible, pairing with "a colleague who has not had this access (4-eyes principle)".
Clause 6.4 is the expensive one for vision systems: objects used for the final test should not have been used to train or validate, unless features are independent. A physically segregated set of vials or tablets, inventoried against test run identifiers, is where the cost lands. The discipline of building the hold-out set applies, but none of these controls come with it.
The "neither" rows are where a private perimeter changes the answer, for a reason unrelated to data residency: you cannot show that no copy of your hold-out set left your control once it was uploaded for fine-tuning, cannot audit a provider's staff separation, and cannot produce a configuration hash for parameters you never held. Annex 11 supplies the lever: its 7.1 keeps the regulated user "fully responsible" for a vendor-supplied system, and 7.4 requires documentation be "accessible and can be explained from their facility".
So for the critical-path classifier, keep the weights, the test repository and the inference runtime inside the perimeter, and record the frozen model's hash in change control (cloud vs on-premise AI security and cost covers the general trade). For the permitted non-critical LLM, clause 10.5 wants review records, and a self-hosted endpoint logs prompt, output, model hash and reviewer against a batch record.
| Clause | What it asks for | Artefact for an inspector | Who can hold it |
|---|---|---|---|
| 3.1 to 4.3 | Intended use, sample space, subgroups, criteria set before testing | Signed intended use, subgroup and criteria | In-house |
| 5.1 to 5.5 | Stratified test set, verified labels, justified exclusions | Characterisation and label verification | In-house |
| 6.1 to 6.3 | No test data in training, access control and audit trail, no copies outside | Repository access log and audit trail | Supplier, and it will not show you |
| 6.4, 6.5 | Final-test objects unused in training, staff separation or 4-eyes | Object inventory, role separation record | Neither, if the supplier trained it |
| 7.2 to 8.1 | Approved test plan, retained records, contributing features captured | Plan, results, SHAP or LIME output | In-house if you hold the weights |
| 9.1 to 10.5 | Confidence logged, change and config control, drift metrics | Confidence log, change records, config hash | Config hash is neither, when hosted |
What the FDA has already enforced without an AI rule
One regulator has already written AI into an enforcement document. Warning Letter 320-26-58, dated 2 April 2026 and issued by CDER after an inspection of 28 to 30 October 2025, carries a section headed "Inappropriate Use of Artificial Intelligence in Pharmaceutical Manufacturing".
The first finding: the firm "used AI to create drug product specifications, procedures, and master production or control records... If you use AI as an aid in document creation, you must review the AI generated documents to ensure they were accurate and actually compliant with CGMP. Your failure to do so is a violation of 21 CFR 211.22(c)." The second is about reliance, not drafting: no process validation before distribution as required under 21 CFR 211.100, which the firm explained by saying "the AI agent you used, never told you it was required." The remedy requires that any output or recommendation from an AI agent "must be reviewed and cleared by an authorized human representative of your firm's QU".
Keep it in proportion. The recipient is a small homeopathic drug manufacturer that had committed to cease production, and the same letter cites insanitary conditions and untested components under 21 CFR 211.84(d). The AI heading is novel, not the severity. Mapped onto the draft annex, the first finding lands on clause 10.5 review records, the second on clause 3.1 and its demand that the intended use state what the tool is not qualified to tell you.
The FDA draft guidance reaches manufacturing through its drug product life cycle definition, while excluding drug discovery and operational efficiencies that do not affect patient safety or quality, and its model risk definition pairs model influence with decision consequence. The EU draft draws its line around a category of model; the FDA framework draws it around a decision.
| Text | Date | Status | Position on generative or probabilistic models |
|---|---|---|---|
| Draft EU GMP Annex 22 | 7 July 2025, closed 7 Oct 2025 | Not adopted, absent from EudraLex | Excluded from critical applications, permitted non-critical with a human |
| FDA draft guidance, docket FDA-2024-D-4689 | January 2025 | Still draft | No model-class exclusion, a 7-step credibility assessment |
| FDA and EMA Guiding Principles of Good AI Practice | January 2026 | Non-binding, 2 pages, 10 principles | No model-class exclusion, validation proportionate to use |
| EMA reflection paper EMA/CHMP/CVMP/83833/2023 | 9 September 2024 | Adopted reflection paper | Close human supervision, generative models prone to erroneous output |
The gap assessment that survives the final text
Clause numbers will move in the final text; section names are less likely to, so key the evidence record on names. The template below is ours, not a schema any regulator publishes, and every field keys to a clause cited above.
model_release:
intended_use: # clause 3.1
task: "classify filled vial images as accept or reject"
input_sample_space: "line 3 camera, 4 formats, rare variations"
limitations: "no amber vials, no wet-label surfaces"
sme_approver: "process SME, before acceptance testing"
subgroups: # clause 3.2
decision_output: [accept, reject]
defect: [particulate, crack, fill_low]
acceptance_criteria: # clauses 4.1 to 4.3
metrics: [sensitivity, specificity, precision, f1] # per subgroup
baseline_process_performance: "measured manual inspection rate"
test_data: # clauses 5.1 to 5.6
selection: "stratified over all subgroups, rare variations included"
labelling_verification: "two independent experts, adjudicated"
generated_by_generative_ai: false
test_data_independency: # clauses 6.1 to 6.5
repository_access_control: true
audit_trail: true
copies_outside_repository: none
physical_objects_separated: true
staff_separation_or_4_eyes: "reviewer had no test data access"
test_execution: {plan_approved_by: "SME", all_test_data_used: true} # 7.1-7.4
explainability: {method: shap, feature_review_approved_by: "QA"} # 8.1, 8.2
confidence: # clauses 9.1, 9.2
score_logged_per_decision: true
undecided_band: "below threshold, route to manual"
operation: # clauses 10.1 to 10.5
change_control_id: "CC-2026-0431"
configuration_hash: "sha256 of frozen weights and runtime image"
input_drift_metrics: defined
human_review_records: retainedSix moves are safe whatever the final wording says. Inventory every model touching a GMP decision, including ones bought as a feature of something else. Classify each with the three-impact test. Freeze and hash the weights for anything critical. Separate the test repository behind its own access control and audit trail. Write the intended use and acceptance criteria before the next retraining. Log a confidence score per decision with an undecided band, which clauses 9.1 and 9.2 ask for without naming a number, then point your monitoring for drift in the input data at 10.3 and 10.4.
One drafting detail to re-read when the final text lands: clause 4.3 cross-references "Annex 11 2.7" for the no-decrease principle, but in the co-published Annex 11 draft 2.7 is Security and no risk increase is 2.8. Which is why the record keys on section names.
The decision worth making this week is small. For the models on your critical list, can you freeze the weights, hash them, and reproduce an output twice? If not, because inference runs behind someone else's endpoint, that is the gap to close first. The rest of the annex is evidence you can assemble later. Custody of the model is not. For the wider picture, see our AI for business pillar.
FAQ
Quick answers to the questions this post tends to raise.
