Migrating a SaaS Platform from Cloud LLM APIs to Self-Hosted Models

The product worked and the prompts were tuned; that was never the problem. What did not improve was the unit economics: token spend tracked usage almost linearly, so every new account carried a variable cost no application engineering removed. At the client's growth rate the API bill was on track to double within a year.

The second forcing function was harder. Two enterprise deals with regulated banks had stalled in vendor risk review, and the objection was not AI; it was an architecture where customer data left the jurisdiction and landed with a subprocessor the bank did not choose. Self-hosted inference removes a row from the subprocessor list, and for this client that row was worth more in the sales cycle than the token savings.

We built the evaluation harness before touching the workload, because a migration is a claim about equivalence and the claim needs ground truth. We sampled production traffic, stratified it by task and difficulty, and built a golden set of roughly 2,000 items with human-verified labels. Scorers stayed deterministic wherever the task allowed: exact match, schema validation, field-level F1. Where a judge model was unavoidable we calibrated it against human labels and reported agreement, because an uncalibrated judge is a confident random number generator.

The baseline run measured the incumbent API against that golden set, and that number became the bar the migration had to clear. Then we classified the traffic by difficulty, and the distribution came out the way it usually does: 92 percent of calls were narrow work, extraction into a schema, classification, routing, and short summarization, with a thin tail of genuinely difficult requests. The client had been paying frontier prices on every request to serve that tail.

Particula-Classify assigns the tier before generation, deliberately as a small-model decision, because using the large model to decide whether the large model is needed defeats the purpose. The bulk tier is a fine-tuned Qwen 3 32B served with vLLM; we fitted Qwen 3, Llama 3.3, and Mistral Small candidates against the harness on rented H100s before committing, and the tuned 32B won on cost against quality. Structured output goes through Particula-JSON with schema-constrained decoding, so valid JSON is a property of the decoder, not something you retry into existence.

The hard tail kept a route out. The router sends genuinely difficult requests to a commercial API where the tenant's data policy permits it, and that permission is per-tenant configuration, not a global build flag. This is the honest version of the architecture: cloud optional, not cloud never. The client's regulated tenants run fully local and accept the ceiling; commercial tenants keep the fallback for the residual traffic that needs it.

We standardized hard on the stack below. Migrations fail on operational surface area far more often than on model quality, and every bespoke component is something that gets debugged at three in the morning by someone who did not build it.

Two omissions were deliberate. We did not build a custom batching or scheduling layer over raw generation calls: continuous batching and paged attention are solved, and a homegrown queue underperforms while costing a maintainer. And we did not accept closed-weight models behind a private endpoint as the migration target. If you cannot hold the weights on your own disk you have changed a billing model, not an architecture, and the deprecation risk that motivated the move is untouched.

The inference tier runs in its own network segment with no default egress. Weights were pulled once, verified by digest, and stored on disks the client controls. Images are pinned by digest rather than tag, so a rebuild in month eight produces the server that passed review in month two. Where the commercial fallback is enabled for a tenant it is the only permitted outbound path, logged per request, and the default for a new tenant is off.

Logging is where teams get this wrong. An eval harness is hungry for production traffic, which is full of personal data. We redact at ingestion rather than after storage: identifiers stripped or tokenized, the retained sample limited to what the scorers need, retention windows per tenant with real deletion instead of a soft flag. Audit records capture which tier and model version served each call, which is what an auditor asks for and what a regression investigation needs anyway.

The regulatory win was architectural rather than procedural. Self-hosted inference removed a cross-border transfer and a subprocessor from the client's data map, which changed what their data processing agreement has to say. The answer to where the data goes is now a diagram with no external arrows, or exactly one that is labeled and switchable off per tenant, and that is the diagram their sales engineers now open in security reviews.

The engagement ran twelve weeks, sequenced so the existing system kept serving every request while the risky work happened beside it. Weeks one through three were the harness: traffic sampling, task taxonomy, golden set construction, scorer implementation, and the baseline run against the incumbent API. Weeks four through eight covered candidate fitting, fine-tuning, and serving. Quantization was treated as a quality decision rather than a memory trick: we ran the harness at each precision and kept FP8, the smallest level that held parity.

The remaining weeks were shadow, canary, and ramp. Shadow mode mirrored live traffic to the self-hosted tier and scored both outputs without serving them, producing a parity report on the real distribution, not the golden set alone. Cutover was gated on that report against the threshold agreed in week one, never on a judgment call made under deadline. The ramp proceeded in steps with automatic fallback on error rate or latency regression; the fallback never fired.

The fine-tuned tier scored 99.2 percent of the incumbent API's pass rate on the golden set, and the shadow-mode parity report on live traffic came in above the agreed threshold before cutover. Post-cutover, the client's own customer-facing quality metrics showed no regression, which is the number their product team actually watches. Routing settled at 96 percent of traffic on the self-hosted tier, with the remaining hard tail going to the commercial API for tenants whose policy permits it.

The cost line moved the way the arithmetic said it would. A bill approaching $28,000 a month in variable token spend became under $7,000 a month all-in: bare-metal GPU rental, the residual fallback API spend, and the operational fraction of an engineer, a reduction of roughly 75 percent at flat volume. More important than the level is the shape: inference is now a fixed cost, so margin improves as the client grows instead of tracking someone else's price list. p95 latency on the bulk tier also came down by roughly a third, since a right-sized 32B on dedicated hardware answers faster than a rate-limited frontier endpoint.

The procurement effect was the quieter win. With inference EU-resident and the subprocessor row gone from the data map, the security-review objection that had stalled the two bank deals disappeared, and both reviews subsequently cleared. The evaluation harness stayed behind as permanent infrastructure: every prompt change and model update re-runs it, which means the client can keep migrating models, in either direction, without ever again arguing from impressions.