Air-Gapped LLM Deployment for a European Power Utility

The operator works under a rule most AI vendors have never had to design around: no route to the internet from the operational segment, enforced by policy, by regulator expectation, and by physical separation. That rule is not negotiable at the engineering level, and it is not satisfied by a private cloud region or a vendor attestation. Every SaaS assistant, every managed model API, and every tool that checks a license server at startup failed the first pass of their security review, which is why the work had stayed manual for years.

The knowledge existed and most of it was written down, but it sat across scanned binders, a document system nobody enjoyed using, and a share full of folders named some variant of final. A shift engineer verifying that a permit matched the current procedure revision could spend twenty minutes assembling the answer from three sources. That is the gap this project closed, without asking anyone to weaken the boundary.

We deployed four tiers on a single chassis. The ingest tier watches an internal file share, runs Tesseract OCR over scanned material, transcribes control-room audio with Whisper, chunks by document structure rather than fixed token windows, and writes text and embeddings to PostgreSQL. The inference tier is a vLLM process serving Qwen 2.5 72B in FP8 across both GPUs under tensor parallelism, with our specialist models resident alongside it for extraction and routing. The application tier is Lumen in its air-gap configuration plus a few FastAPI services. The fourth tier is local operations: the eval harness, metrics, and the audit log.

Retrieval is where these systems fail, so that is where the design effort went. Procedures and permits are structured documents with numbered clauses, revision markers, and equipment tags, and that structure is worth more than any embedding model. We parse it, index it, and make every answer carry a citation down to clause and revision. An answer without a traceable source is unusable where a wrong valve number is a safety event, so Lumen is configured to decline rather than guess when retrieval comes back thin.

We deliberately did not run Kubernetes. A two-GPU appliance behind an air gap has nothing to schedule, and every component of that control plane becomes another thing to patch through a signed package process. Docker Compose and a written runbook are the right altitude. We also skipped fine-tuning as the opening move: retrieval plus a locked output schema covered what the site asked for, and a fine-tune is a model you must re-train, re-evaluate, and re-transfer every time the corpus moves.

Model choice at this size is a VRAM budget spent deliberately. A 70B-class model at FP16 is roughly 140 GB of weights before any KV cache, which no compact server holds. On the two 80 GB cards we chose FP8 over 4-bit because it sits closer to lossless and the kernels are mature, leaving comfortable headroom for KV cache and concurrency. We evaluated Qwen 2.5 72B and Llama 3.3 70B against the site's own question set during the outside build; Qwen won on structured extraction of equipment tags and long-context retrieval, which are exactly the failure modes that matter here.

What we refused to do was quantize and hope. Degradation at this scale shows up first in long-context retrieval and structured output, not in conversational quality, so a model that still chats well can quietly start dropping equipment tags. The FP8 build was re-run against the full site eval set before it went anywhere near a crew, with the fallback option of dropping to a 30B-class model held open until the numbers came in. A smaller model that answers correctly beats a larger one that sounds correct.

The boundary is the product. The operational network has exactly one inbound path, an offline package, and no outbound path at all. Release bundles are assembled outside: container images exported as OCI tarballs, model weights with per-file checksums, database migrations, configuration, an SBOM, and a manifest. The manifest is hashed and signed with a key held offline. The bundle crosses on scanned removable media through the transfer enclave the operator already ran for firmware and vendor patches, so no new process had to be invented or approved.

Inside, nothing installs until the signature verifies against a pinned public key and every file hash matches the manifest. On any mismatch the installer refuses and leaves the running version untouched. Each install writes an immutable record of bundle identity, signer, operator, and timestamp, which is what makes the system auditable: the operator's internal auditor can reconstruct which model weights and which container digests were answering questions on any given date. Rollback is a previous image tag already on disk, not a rebuild.

The remaining controls are unglamorous on purpose. Authentication binds to the site directory, roles map to document libraries so a contractor cannot retrieve restricted procedures, prompts and responses log to PostgreSQL under the site retention schedule, and the inference host carries no outbound firewall rule at all rather than a permissive one that merely goes unused. We stripped telemetry at build time instead of switching it off in configuration, because a disabled feature is one edit away from being enabled again.

The engagement ran fourteen weeks, sequenced so the risky work happened outside the boundary. Weeks one and two were boundary and sizing review: document classes, the network diagram, the transfer path and who was authorized to use it, hardware lead times, and the sign-off chain. Weeks three through five were an outside build on identical hardware, where we assembled the stack, tuned retrieval against the corpus sample the operator could release, and constructed a 400-question evaluation set with the site's own senior engineers, scored on expected sources rather than expected prose.

Weeks six and seven covered transfer and install, including full ingest of the 28,000-document corpus inside the fence. A supervised pilot with the control-room crew ran four weeks, every answer carrying citations and the engineers grading a daily sample. The final stretch was handover, and it was not documentation theatre: the operator performed the second package transfer themselves while we watched, and we ran a rollback drill on the production hardware. A site that has never rolled back is a site that cannot roll back.

Cutover was gated on eval parity, not on the calendar. The inside build had to match the outside build on the 400-question set before access widened beyond the pilot crew, and it did.

In the final two weeks of the pilot, 94 percent of graded answers were rated correct with correct clause-and-revision citations by the site's own engineers. The system declined roughly seven percent of questions where retrieval came back thin, and the operator counts those declines as correct behavior: a refusal is recoverable, a confident wrong valve number is not. The cross-referencing task that used to take a shift engineer twenty minutes, checking a permit against the current procedure revision, now resolves in under a minute with the source clauses on screen.

The evaluation harness is what makes the system maintainable with zero telemetry. Every model change, prompt change, and corpus reindex re-runs the versioned 400-question set locally and prints a diff against the previous build, which is the only thing standing between a routine update and a silent regression. Runtime metrics scrape to a time-series store on the same box, request and citation logs sit in PostgreSQL, and the dashboard is read by the site's own team. There is no crash reporter and no usage beacon.

Since handover the operator has carried out two update cycles through the signed package process without us in the room, which was the real acceptance test. The named package owner, the rehearsed rollback, and the local eval harness are what keep an air-gapped system from becoming the most trusted and least maintained thing on the network. That, more than the model, is what this deployment delivered.