OpenAI Privacy Filter vLLM: Self-Host It and Decode Spans

The card describes an Apache 2.0 model with 1.5B parameters total and 50M active: 8 transformer blocks and a sparse mixture of experts with 128 experts and top-4 routing per token. The vLLM PR calls it a gpt-oss-style MoE encoder with the language-model head replaced by a 33-class score head.

Attention is bidirectional and banded with a band of 128, so each token sees an effective window of 257 tokens including itself, inside a 128,000-token context. Per-token scores are local evidence; span structure has to be enforced on top.

The output space is eight labels, each expanded into four BIOES tags (begin, inside, end, single), plus background O. That is 32 plus one, 33 classes, and config.json orders them alphabetically by label:

Class 0 is O. This order matches opf's v2 label space, which is what lets the client below reuse opf's helpers against vLLM's rows.

The label policy is fixed. Section 4.2 of the card: "Privacy Filter does not support configuring label policies dynamically at runtime; instead, changing policies requires further fine-tuning of the model." If your data protection officer wants employee IDs masked, no request parameter does that.

Pin v0.33.0. It was released on 2 November 2026 and is the latest final vLLM release at the time of writing. The first release with the model, v0.25.0, is not an acceptable pin for this guide. CVE-2026-93592 (GHSA-25q3-v2hm-8vpf, rated High, published 3 September 2026) lets a negative token id in the token-id form of input on /v1/embeddings or /pooling kill the engine, and 0.28.0 added the vocabulary check that blocks it. That check was incomplete. GHSA-q9v9-qpjp-fc24 (rated High, published 2 November 2026, affecting 0.28.0 up to but not including 0.33.0) shows it only runs when a tokenizer is loaded, so a server started with --skip-tokenizer-init accepts an out-of-range token id and one unauthenticated request crashes the shared engine for every tenant, on /v1/completions and equally on /v1/embeddings and /pooling. The command below does not set that flag, but the client sends pre-tokenized ids, the mode the flag exists for. Do not add it, and pin 0.33.0 so the bound holds either way. The full advisory list for the server belongs in our vLLM inference server hardening guide.

pip install "vllm==0.33.0"

vllm serve openai/privacy-filter \
  --revision 7ffa9a043d54d1be65afb281eddf0ffbe629385b \
  --runner pooling \
  --pooler-config.task token_classify

Each flag earns its place:

You do not manage the Transformers version. The model arrived in Transformers v5.6.0 (released 22 April 2026), and vLLM v0.33.0 pins transformers >= 5.16.1, < 5.21.0 in its own requirements, so the 5.6.0 floor only matters if you also run the model standalone in Transformers.

For AMD estates: the v0.33.0 parity test for this model is still skipped on ROCm, pointing at a MoE workspace allocation issue (#48510) that was closed on 20 October, with the fix PRs (#48179, #65318) still open on 6 November. Treat ROCm as unvalidated for this model on v0.33.0.

The request at v0.33.0 takes model, input (a string, a list of strings, a list of token ids, or a list of lists), task, and use_activation, plus truncate_prompt_tokens, add_special_tokens and encoding_format. The response is a data list with one entry per input, and each entry's own data field holds the [T, 33] scores. By default the token classifier head applies softmax over the last dimension, so you get probabilities. With use_activation: false you get raw logits.

Three things are absent: token strings, character offsets, and span decoding. The v0.33.0 model file's forward pass returns self.score(hidden_states), one linear layer from 640 to 33, with no Viterbi, no CRF and no BIOES transition logic.

Argmax picks each token's best class independently, so nothing stops O, I-secret, E-secret (an inside tag with no begin) or B-private_email, E-private_phone (a span that changes label mid-way). Both are invalid BIOES. A span builder either drops them, losing a secret, or repairs them with heuristics, which is the decoder's job. The card's case for Viterbi is span coherence on noisy, mixed-format text: a constrained pass makes invalid transitions impossible and picks the best valid label path for the whole sequence.

Here is where each path leaves you:

The only path that matches the card's description of how spans are made is the one that does not ship as a package. That is the gap to close.

The fix is to keep vLLM for the forward pass and run opf's decoder in your client. opf's decoding helpers import torch and tiktoken but not the model or its Triton kernels, so the client does not need a GPU or the weights. Install opf for its decoder only, pinned to the current head of main:

pip install "opf @ git+https://github.com/openai/privacy-filter@f7f00ca7fb869683eb732c010299d901457f19c3"

The client tokenizes with o200k_base exactly as opf does, sends the token ids to /pooling with use_activation set to false, applies log_softmax to the logits the way opf's own predict_text does, runs ViterbiCRFDecoder with a calibration file, and converts token spans to character spans with opf's own helpers:

import requests
import tiktoken
import torch

from opf._common.label_space import NER_CLASS_NAMES_BY_CATEGORY_VERSION
from opf._core.decoding import (
    ViterbiCRFDecoder,
    resolve_viterbi_biases_from_calibration_path,
)
from opf._core.sequence_labeling import build_label_info
from opf._core.spans import (
    decode_text_with_offsets,
    labels_to_spans,
    token_spans_to_char_spans,
    trim_char_spans_whitespace,
)

POOLING_URL = "http://pii-filter.internal:8000/pooling"
ENC = tiktoken.get_encoding("o200k_base")  # original/config.json "encoding"
LABELS = build_label_info(NER_CLASS_NAMES_BY_CATEGORY_VERSION["v2"])  # same order as id2label
DECODER = ViterbiCRFDecoder(
    label_info=LABELS,
    **resolve_viterbi_biases_from_calibration_path("/etc/pii/viterbi_calibration.json"),
)


def detect_spans(text: str) -> list[tuple[str, int, int]]:
    if not text:
        return []
    token_ids = ENC.encode(text, allowed_special="all")
    resp = requests.post(
        POOLING_URL,
        json={
            "model": "openai/privacy-filter",
            "input": token_ids,
            "task": "token_classify",
            "use_activation": False,  # raw logits; opf applies log_softmax itself
        },
        timeout=30,
    )
    resp.raise_for_status()
    logits = torch.tensor(resp.json()["data"][0]["data"], dtype=torch.float32)
    if logits.shape != (len(token_ids), 33):
        raise ValueError(f"unexpected score shape {tuple(logits.shape)}")

    labels = DECODER.decode(torch.log_softmax(logits, dim=-1))
    token_spans = labels_to_spans(dict(enumerate(labels)), LABELS)
    decoded, starts, ends = decode_text_with_offsets(token_ids, ENC)
    char_spans = trim_char_spans_whitespace(
        token_spans_to_char_spans(token_spans, starts, ends), decoded
    )
    return [(LABELS.span_class_names[lab], s, e) for lab, s, e in char_spans]

Why token ids rather than a string: when you tokenize, row i of the response is token i of your own token_ids, and opf's offset helpers map that token back to characters. Send a string and the server tokenizes with its own tokenizer, and you lose the guarantee that your offsets line up with its rows. The shape check turns any misalignment into an exception instead of a silently shifted mask.

Five caveats belong in your code review:

The card talks about preset operating points. What ships is one: viterbi_calibration.json (top-level and in original/) holds a single point named default with all six transition biases at 0.0. opf's loader enforces that shape: exactly operating_points, containing exactly default, containing exactly biases with the six keys. Anything else raises ValueError. So keep one calibration file per operating point, which means one per pipeline.

Each bias adds a constant to one class of edge in the decoder's transition table, on top of the per-token log-probabilities:

Disallowed BIOES transitions get a score of minus one billion, so they never win; if no finite path exists the decoder falls back to argmax. The direction matches section 2.3.2 of the card, which says discouraging background and encouraging span starts and continuations yields broader, more contiguous redaction for better recall. For a log or trace pipeline where false positives are the complaint, go the other way: raise background_stay, lower background_to_start, and entering a span gets costlier.

{
  "operating_points": {
    "default": {
      "biases": {
        "transition_bias_background_stay": 0.5,
        "transition_bias_background_to_start": -0.5,
        "transition_bias_end_to_background": 0.0,
        "transition_bias_end_to_start": 0.0,
        "transition_bias_inside_to_continue": 0.0,
        "transition_bias_inside_to_end": 0.0
      }
    }
  }
}

The two non-zero values are placeholders for the start of a sweep, not recommendations. The card publishes no figures for what a non-zero bias does on log text, and it does not say which operating point its benchmark tables used. Sweep both in small steps against the per-class eval below, keep the point that meets your recall floor on the classes you cannot miss, and name the file after its pipeline. The card lists "relying on default operating points without validating them on the target distribution" as misuse.

The public issue tracker shows how the model fails on text people actually fed it. The first five rows are GitHub issues, all open on 6 November 2026; the last is the card's own failure list:

The #34 evidence came from the argmax path, so it does not show what the Viterbi path or a bias change does to UUIDs. That is why the first fix is deterministic. If a UUID in a trace id is never a credential in your schema, a post-filter that drops secret and account_number spans whose exact text is a UUID costs nothing and does not depend on calibration:

import re

UUID = re.compile(r"[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}")
GUARDED = {"secret", "account_number"}


def drop_benign_ids(text: str, spans: list[tuple[str, int, int]]) -> list[tuple[str, int, int]]:
    return [
        (label, s, e)
        for label, s, e in spans
        if not (label in GUARDED and UUID.fullmatch(text[s:e]))
    ]

Keep the allowlist short and specific to formats you generate yourself. Every pattern on it is a class of string that will never be masked, so it belongs under the same review as the calibration file.

The deeper problem for log text is in the card's Table 5, which measures precision and recall on English synthetic text with and without a category clue next to the value:

Per class, bare values are weaker still: private_address recall 0.214, private_date 0.220, private_phone 0.318, secret 0.361. Log lines and JSON fields look exactly like that, so for log redaction this row sets your recall floor.

Build that eval with opf's own harness. It runs the local opf model, not vLLM, so it is your calibration bench. Point it at the opf-native original/ checkpoint downloaded at the pinned revision, and at a few hundred labelled lines of your own text:

opf eval /data/pii/holdout.jsonl \
  --checkpoint /models/privacy-filter-7ffa9a0/original \
  --device cuda \
  --eval-mode typed \
  --per-class \
  --viterbi-calibration-path /etc/pii/viterbi_calibration.json \
  --metrics-out /data/pii/metrics-logs-op1.json

--eval-mode typed with --per-class reports precision and recall per label, the only view that shows whether secret is caught. Run it with the shipped all-zero file and with each candidate point, keep each --metrics-out file beside its calibration file, and repeat the final check through the vLLM client, since the two backends are close rather than identical.

Operating points move the boundary between "mask" and "leave" within the eight labels. They cannot add a label or change what counts as one. When the mismatch is policy, fine-tuning is the documented path, and the card's Table 2 shows how large that effect can be. On the SPY dataset, which labels only the question author as PII and has its own account-number types, token-level results move like this:

The base model's recall on SPY is already 0.795; its precision is 0.414. It finds things the dataset does not consider PII. That is a policy disagreement, and 1% of the training split closes most of it.

The tooling is opf train <train.jsonl> --output-dir <dir>, with flags including --validation-dataset, --label-space-json, --epochs, --batch-size and --learning-rate. The dataset is shared with opf eval: a text field plus character-offset spans. A custom label space is a JSON file with a category_version and a span_class_names list starting with O.

Here is the catch for a vLLM fleet. The training runner writes an opf-native config.json, a copy of the base opf config with model_type privacy_filter, no architectures field and no id2label. vLLM resolves the model class from architectures, so it cannot serve that directory as-is. Neither opf at the pinned commit nor the Transformers v5.6.0 model directory ships a conversion script. The runner also does not copy viterbi_calibration.json, so a fine-tuned model starts with no calibration at all.

So a policy change by fine-tuning means serving the result with opf, or writing and owning a converter. opf at the pinned commit has no batched redaction (PR #28 is open) and no per-span confidence scores (PR #8 is open). Budget for either before you start labelling.

Self-hosting is the reason to pick this model in a regulated estate: the weights, the decoder and the fine-tuning loop all run inside the perimeter under Apache-2.0. The same fact moves three things from the vendor's column to yours.

First, the decoder is now your control. Redaction behaviour is defined by client code you run, not by the model, so it needs the same pinning, review and change control as any other data-protection component.

Second, the operating point decides what reaches your log store, your index or your LLM. Version the calibration file with the pipeline code and its eval output. In the redaction layers we build, the file, its metrics and the pinned revision ship together or not at all.

Third, the upstream. As of 6 November 2026 both repos last changed on 22 April. Only PR #1 was ever merged. There are 15 open issues and 18 open community PRs (the repo counter shows 33 because it adds them). The maintenance question (#39) has no reply, the PyPI question (#33) got a same-day roadmap promise in June and no release since, and the PR that would let opf pin its default checkpoint download (#16) is open, so opf floats to the Hugging Face main branch by default. The card calls the release a preview. That predicts nothing about OpenAI's plans, but it settles who maintains your copy today: you do.

The card is explicit about scope: "a redaction and data minimization aid, not an anonymization, compliance, or a safety guarantee", with a high-risk caution covering medical, legal, financial, HR, education and government workflows. For multilingual estates, per-language F1 runs from 0.914 (Dutch) to 0.934 (English) on PII-Masking-300k and from 0.758 (Hausa) to 0.933 (Portuguese) on the card's synthetic set; read those as a ceiling on benchmark text. The model is one layer; the controls around it are in how to prevent data leakage in AI applications.

The ownership checklist is short:

The wider picture of securing self-hosted AI, from inference servers to data controls, is in the AI security pillar.