Ray Cluster Authentication: RAY_AUTH_MODE and Five CVEs

CVE-2023-48022 was published on 28 November 2023 and describes remote code execution through the job submission API. It carries a vendor note disputing relevance, on the grounds that Ray is not intended for use outside a strictly controlled network environment. On its own terms that argument holds: if the only supported deployment is a private segment, an unauthenticated job API is a documented property rather than a defect. The record is disputed, not rejected, and it has stayed live since.

The record itself now undercuts that argument. Its vendor note adds, parenthetically, that within that environment customers at version 2.52.0 and later can choose to use token authentication, and its reference list includes Ray's token authentication page. The advisory once answered with a network diagram now points at a feature.

What the feature is worth is stated exactly in Ray's security overview: token authentication is not an alternative to deploying Ray clusters in a controlled network environment, rather it is a defense-in-depth measure that adds to network-level security. Two controls, both required.

The 2.58.0 documentation still says authentication is disabled by default in Ray 2.52.0, and that Ray plans to enable it by default in a future release. Enablement is an operator action, not something an upgrade hands you.

Five identifiers, three classes of problem: one architectural note that no patch closes, two browser-reachable control-plane bugs that upgrading closes, and two Ray Data deserialization bugs on the data path that token authentication does not touch at all.

Both control-plane bugs show the shape of the defence that failed. In CVE-2025-62593 the guard on the job endpoints was a helper returning whether the User-Agent header starts with Mozilla, described in its own docstring as a very weak heuristic. The fetch specification lets a page modify that header, so with DNS rebinding a developer visiting a web page in Safari or Firefox became remote code execution on a cluster that was never routable from the internet. In CVE-2026-27482 a middleware rejected browser-origin requests only when request.method was POST or PUT, leaving DELETE uncovered. PR #60526 replaced the deny-by-method decorator with one allowing only an explicit set of methods. If you reconcile feeds, note that the CVE record carries CWE-396 while the GitHub advisory carries CWE-306 for the same defect.

The two Ray Data bugs are a different animal. CVE-2026-41486 covers three globally registered Arrow extension types, ray.data.arrow_tensor, ray.data.arrow_tensor_v2 and ray.data.arrow_variable_shaped_tensor: reading a Parquet file carrying one calls __arrow_ext_deserialize__ on the field's metadata bytes, which reached cloudpickle.loads(), so execution happened during schema parsing, before any row data was read. CVE-2026-57516 covers _default_decoder() in the WebDataset datasource, reachable through ray.data.read_webdataset(). Neither is gated by RAY_AUTH_MODE, because both are reached by handing a file to a cluster you are already authorised to use. On scanning those files rather than trusting them: scanning a pickle file does not make it safe.

Both fixes work by inverting a default, which creates a trap. The 2.55.0 patch parses Arrow extension metadata as JSON and falls back to cloudpickle only behind an opt-in environment variable. Its in-code comment says cloudpickle deserialization is an opt-in for files written by Ray 2.49 to 2.54 but must not be used with untrusted Parquet, and the error a reader hits names the variable to set. Anyone with a warehouse of tensor Parquet from before 2.55.0 has a reason to set it. The three switches to keep unset:

# Ray >= 2.56.0: WebDataset pickle and torch decoding is refused unless you opt in.
# Leave this UNSET. Setting it re-enables the CVE-2026-57516 code path.
# RAY_DATA_WEBDATASET_ALLOW_UNSAFE_DESERIALIZATION=1

# Ray >= 2.55.0: Arrow extension metadata is parsed as JSON, not cloudpickle.
# Leave this UNSET. Setting it re-enables the CVE-2026-41486 code path,
# and it is the only way to read tensor Parquet written by Ray 2.49 to 2.54.
# RAY_DATA_AUTOLOAD_CLOUDPICKLE_TENSOR_METADATA=1

# Writer side: cloudpickle serialization of Arrow extension metadata,
# opt-in since 2.55.0.
# RAY_DATA_ARROW_EXTENSION_SERIALIZATION_CLOUDPICKLE=1

env | grep -E 'RAY_DATA_(WEBDATASET_ALLOW_UNSAFE|AUTOLOAD_CLOUDPICKLE|ARROW_EXTENSION_SERIALIZATION)'
# any output here is a finding

The floor: 2.56.0 if you use Ray Data, 2.54.0 if you do not. Landing on 2.54.x while using Ray Data leaves two open at once, because that line sits below the WebDataset floor and inside the Arrow range. Reading a CVE list as a floor rather than a news feed is the exercise we ran on the storage layer in self-hosted vector database hardening; the generic method sits in how to audit an AI framework stack for CVEs.

One environment variable switches the mode. A separate command materialises the secret. They are two independent steps, because ray start will not create a token and does not fall back to open access when it cannot find one.

# Ray 2.52.0 or later. Auth is off unless this variable says exactly "token".
export RAY_AUTH_MODE=token

# ray start does NOT generate a token for you.
ray start --head
# ray.exceptions.AuthenticationError: Token authentication is enabled but no
# authentication token was found. ... To generate a token for local development,
# use `ray get-auth-token --generate`

ray get-auth-token --generate    # writes ~/.ray/auth_token and prints it
ray start --head                 # now starts

That exception fires precisely when the mode resolves to TOKEN, no token is found, and the caller did not ask for creation. ray.init() takes the other branch and generates. ray start --head does not. Failing closed is the right behaviour, and it means the first node you restart after enabling the mode will refuse to come up unless the token is already on disk.

Ray resolves which token a process presents in three steps, highest priority first: the RAY_AUTH_TOKEN environment variable, then a file named by RAY_AUTH_TOKEN_PATH, then ~/.ray/auth_token. Prefer the second. Ray's documentation recommends storing tokens in files and using RAY_AUTH_TOKEN_PATH rather than setting RAY_AUTH_TOKEN directly, to avoid exposing the token to other code that reads environment variables. That advice is load-bearing for the vLLM section below.

The path variable has one behaviour only the internals design document states: if RAY_AUTH_TOKEN_PATH is set but the file cannot be read or is empty, Ray treats this as a fatal misconfiguration and aborts rather than silently falling back. Whitespace is stripped when reading a token from a file. Setting the mode through system_config instead of the environment is rejected with a RuntimeError.

Ray servers accept the token in four places: gRPC metadata as Authorization: Bearer <token>; the same header over HTTP, which the Ray CLI sends; the cookie ray-authentication-token, set by the dashboard after posting the token to /api/authenticate with HttpOnly, SameSite=Strict and a 30-day max age; and X-Ray-Authorization: Bearer <token>, for KubeRay and environments where a proxy strips the standard header. The middleware returns 401 for a missing token and 403 for an invalid one, which is how you tell a distribution failure from a wrong-token failure.

One caveat from the same document: for HTTP, middleware and header injection are not automatically wired up for new services and must be added manually. The aiohttp middleware is explicitly registered on the dashboard head and the runtime env agent. Token authentication is not a blanket property of every HTTP surface Ray grows.

A dashboard that prompts for a token while the mode is disabled was a 2.52.1 defect, not a policy change: the front end opened its token dialog on any 401 or 403 from any API call regardless of mode. The fix merged 6 December 2025 and shipped in 2.53.0 on 20 December 2025.

The single-node instructions do not survive contact with a cluster. Every node and every client needs the mode set in its environment and the same token available to it. Miss either half on any node and that node is either dead or unauthenticated.

For a VM cluster launched with ray up, the documented enablement is two YAML keys. One propagates the secret, the other the mode.

# cluster.yaml (excerpt)

# Mount a locally generated token file to all nodes in the Ray cluster.
file_mounts: {
    "/home/ubuntu/.ray/auth_token": "~/.ray/auth_token",
}

# Set the RAY_AUTH_MODE environment variable for all shell sessions on the cluster.
initialization_commands:
  - echo "export RAY_AUTH_MODE=token" >> ~/.bashrc

Ray's documentation also tells you to run ray dashboard cluster.yaml, which sets up SSH port forwarding for the dashboard port, even if you do not plan to use the dashboard. It doubles as the encryption answer for that port.

On self-managed hardware the same two propagations are manual, and the inline export per node is the documented pattern:

ray get-auth-token --generate

scp ~/.ray/auth_token user@node1:~/.ray/auth_token;
scp ~/.ray/auth_token user@node2:~/.ray/auth_token;

ssh user@node1 "RAY_AUTH_MODE=token ray start --head";
ssh user@node2 "RAY_AUTH_MODE=token ray start --address=node1:6379";

Note what you have taken on: a static secret distributed to every node by hand, with file permissions you have to set.

On Kubernetes there are two separate features and they are worth keeping apart. Token authentication is the 2.52.0 feature: KubeRay v1.6.0 or newer exposes an authOptions API on the RayCluster resource, generates a random token into a Secret, and sets RAY_AUTH_TOKEN and RAY_AUTH_MODE on all Ray containers. On an older operator you create the Secret yourself and set both variables on the Ray containers in the pod spec. The Secret and the client half:

# Older KubeRay operator: create the Secret yourself
kubectl create secret generic ray-cluster-with-auth \
  --from-literal=auth_token=$(openssl rand -base64 32)

# Client side
kubectl port-forward svc/ray-cluster-with-auth-head-svc 8265:8265 &
export RAY_AUTH_MODE=token
export RAY_AUTH_TOKEN=$(kubectl get secrets ray-cluster-with-auth \
  --template={{.data.auth_token}} | base64 -d)
ray job submit --address http://localhost:8265 -- python script.py

Without those two client-side exports the same submit returns HTTP 401 and a client-side error that authentication is required with a missing token. A wrong token takes the 403 branch, reported as an AuthenticationError about an invalid token.

Delegating authentication to Kubernetes RBAC is the separate, later feature. It needs Ray 2.55.0 or newer, and the operator enables it by setting RAY_AUTH_MODE to token and RAY_ENABLE_K8S_TOKEN_AUTH to true on all Ray containers, plus mounting a projected service account token. Access then runs through a custom Kubernetes verb: a Role created with --verb=ray:write on the rayclusters resource, a ClusterRole letting the cluster's ServiceAccount reach the TokenReview and SubjectAccessReview APIs, and a client token from kubectl create token. Under this mode a request carries the identity of a Kubernetes principal instead of a cluster-wide shared secret.

Eight properties, and three of them decide an audit: the token rides in the clear over http, sits in plaintext at ~/.ray/auth_token under file permissions alone, and never expires. Ray's documentation states all three, and warns against exposing a Ray cluster directly to the internet because tokens alone do not protect against network eavesdropping.

Rotation is not a command. ray get-auth-token --generate creates a token only when none exists, so rotating means deleting the file on every node, regenerating, redistributing, restarting the cluster and updating every client. On a shared GPU cluster that is a scheduled maintenance window. Decide the interval before you enable the mode.

The deeper limitation is that one shared bearer token is a gate, not a principal. Every job, notebook and CI runner presents the same string, so the access log cannot say who did anything and revoking one consumer revokes all of them: the same structural problem that makes static API keys break down for autonomous agents. Ray's isolation model closes the loop. If workloads require isolation from each other, use separate isolated Ray clusters, because Ray schedules multiple distinct jobs in one cluster without enforcing isolation between them and implements no access controls between developers.

Ray does support TLS on its gRPC channels, and the documentation treats it as a separate control from token authentication: an addition to network isolation, not a replacement.

Ray is not the automatic multi-node backend for vLLM: with the executor backend unset and a world size above one, the selection starts at mp, and for CUDA with nnodes above one the branch explicitly picks multiprocessing. vLLM rejects nnodes above one with any backend other than mp, uni or external_launcher. Ask for a world size larger than the GPUs on the node without setting either and you get the error naming both choices:

World size (16) is larger than the number of available GPUs (8) in this node.
If this is intentional and you are using:
- ray, set '--distributed-executor-backend ray'.
- multiprocessing, set '--nnodes' appropriately.

If you took the Ray branch, this is the design you inherited. vLLM's security documentation, added 30 July 2026, states that it treats the entire Ray cluster as a single trust domain: any principal able to execute code within it has the same level of trust as the driver or API server process, and vLLM does not attempt to isolate driver-side credentials from worker-side processes. A design statement, not a bug.

The mechanism is one dict comprehension. The default executor, selected when the backend is ray and VLLM_USE_RAY_V2_EXECUTOR_BACKEND is truthy (its resolver defaults to 1), builds the worker environment like this:

# vllm/v1/executor/ray_env_utils.py
def get_driver_env_vars(
    worker_specific_vars: set[str],
) -> dict[str, str]:
    exclude_vars = worker_specific_vars | RAY_NON_CARRY_OVER_ENV_VARS

    return {key: value for key, value in os.environ.items() if key not in exclude_vars}

Everything in the driver's os.environ goes to the workers except worker-specific names and an operator denylist, applied worker-side with setdefault semantics. vLLM spells out the consequence: where operators intentionally scope credentials such as HF_TOKEN, cloud storage keys, registry tokens or internal service tokens to the driver alone, the default propagation copies those credentials into worker environments, and a process on a worker node under the same OS user may then read them from /proc/<pid>/environ on Linux.

The denylist is where the surprise lives. RAY_NON_CARRY_OVER_ENV_VARS is loaded from ray_non_carry_over_env_vars.json under the vLLM config root, defaulting to ~/.config/vllm/. If the file does not exist the set is empty; if the JSON fails to parse, vLLM logs a warning and uses an empty set, so a malformed denylist fails open. The example the documentation gives lists seven names you write yourself:

[
  "HF_TOKEN",
  "AWS_SECRET_ACCESS_KEY",
  "AWS_SESSION_TOKEN",
  "GOOGLE_APPLICATION_CREDENTIALS",
  "AZURE_CLIENT_SECRET",
  "REGISTRY_TOKEN",
  "MY_INTERNAL_SERVICE_KEY"
]

The legacy executor leaks a narrower set rather than none: it uses a prefix allowlist, and HF_ is one of the allowed prefixes, so HF_TOKEN still reaches workers while cloud keys do not. The same denylist file is subtracted from that allowlist, so writing it covers both paths.

vLLM recommends three other measures, and unlike the denylist none of them depends on a file that does not exist by default. Keep credentials out of the driver's os.environ, injecting them through a secrets manager, a mounted file or a short-lived subprocess. Restrict procfs visibility on worker nodes, mounting /proc with hidepid=2 or using a runtime that isolates /proc between pods, and run driver and workers under non-overlapping UIDs. And limit Ray cluster access, because access to the Ray cluster is equivalent to arbitrary code execution.

This is the orchestration layer beneath the serving layer. The server's own API controls belong to hardening the vLLM server; for what runs above Ray, we compared the options in choosing the serving stack on top.

Token authentication does not change where Ray listens, and that surface makes a Ray cluster a segmentation problem rather than a firewall-rule problem.

Nine entries, four of them defaulting to a random port on every node. Worker processes bind 10002 to 19999, which the documentation describes as a maximum of 10,000 workers per node irrespective of CPU count; --worker-port-list can pin an explicit short list instead. You cannot write a tidy allow-list for that surface, which is the argument for putting the whole cluster on its own segment and letting only the three named head-node ports cross a boundary.

The dashboard port is the one with a fixed, documented default. Ray's own documentation demonstrates fingerprinting it with nmap -sV --reason -p 8265 $HEAD_ADDRESS, whose sample output names the aiohttp and Python versions. Run that scan from outside the segment.

Order matters because a mismatched token is a hard failure, not a warning.

In a regulated on-premise estate those steps grow extra requirements. The token is a static secret at rest with no expiry, so it needs a named owner, a canonical copy in whatever vault or HSM the estate already runs, a defined file mode on every node, and a rotation procedure honest about being a maintenance window. Write that into the change record before enabling, not after an audit asks.

The plaintext HTTP header is the sharpest point. If you have to evidence encryption in transit for a specific control, an internal segmentation standard or an open audit finding, the token is not what satisfies it, and shipping it over http inside the perimeter puts a long-lived credential on the wire for anything with span-port or sidecar visibility. The reviewable answer is mTLS or an overlay carrying the traffic with the token riding inside it. The vLLM propagation is the other place on-premise beats managed: you own both halves, so keeping the secret out of the driver's environment, writing the denylist file and mounting /proc with hidepid=2 are changes you make rather than tickets you file.

The last consequence is capacity, not security. Ray does not enforce isolation between jobs in one cluster, so one cluster is one blast radius and one data classification. Separate clusters per classification is Ray's documented answer and it costs GPU utilisation: a trade to make in a design review, not during an incident.

Run this against the cluster this week. Four checks, no new tooling:

# 1. What version is actually running
ray --version

# 2. Is the mode set, on the head AND on every worker
echo "$RAY_AUTH_MODE"          # anything that is not exactly "token" means disabled

# 3. Who can reach the dashboard from outside the segment
nmap -sV --reason -p 8265 $HEAD_ADDRESS

# 4. Token file permissions, on every node that has one
ls -l ~/.ray/auth_token

If check two prints an empty line on any node, that node has no authentication, whatever the manifest that was supposed to set it says.