Skip to content

Security

This document provides security guidance for deploying and operating the vLLM Hardware Plugin for Intel® Gaudi®.

Upstream vLLM security guidance

The vLLM Hardware Plugin for Intel® Gaudi® is a hardware plugin that runs on top of the vLLM serving engine. It shares the same architecture, deployment model, and threat surface as upstream vLLM.

The vLLM Hardware Plugin for Intel® Gaudi® follows the same security assumptions as vLLM. Before deploying this project, read and follow the upstream vLLM security documentation:

These recommendations apply directly to Intel® Gaudi® deployments. In particular:

  • Network isolation between nodes
  • Running behind a reverse proxy
  • Restricting endpoint exposure
  • Avoiding development mode in production
  • Loading models and adapters only from trusted sources

General deployment recommendations

The following advice complements the upstream guidance and is worth reviewing before you expose a deployment:

  • Run behind a trusted network boundary. vLLM does not authenticate most of its endpoints by default, and inter-node communication is insecure by default. Place your deployment on an isolated network and put it behind a reverse proxy or API gateway that enforces authentication and TLS. Do not expose the server directly to the public internet.

  • Restrict file and directory permissions. Limit access to model weights, cache directories (for example the Hugging Face cache and any recipe/graph cache), calibration measurement files, and configuration files to the user account that runs the vLLM process. A user who can write to a cache directory that vLLM loads from may be able to achieve arbitrary code execution. Do not share these directories with untrusted users or mount them from untrusted storage.

  • Load only trusted models and adapters. Model files, quantization/calibration artifacts, and LoRA adapters can execute code when loaded. Only use artifacts from sources you trust, and avoid enabling runtime/dynamic adapter loading on untrusted input.

  • Never enable development mode in production. Do not set VLLM_SERVER_DEV_MODE=1 on a production deployment, as it exposes additional debug endpoints.

  • Protect secrets. Keep tokens, such as HF_TOKEN, and other credentials out of source control and out of container images. Provide them through environment variables or a secrets manager, and scope them to the minimum required access.

  • Keep dependencies up to date. Track upstream vLLM and Intel® Gaudi® software releases and apply security fixes promptly.

Reporting a vulnerability

To report a security vulnerability in the vLLM Hardware Plugin for Intel® Gaudi®, follow the process described in the project's Security Policy.

For vulnerabilities in the upstream vLLM engine, use the vLLM vulnerability reporting process.