Manually Load LORA

Download LoRA adapters

Download a LoRA adapter from HuggingFace to your persistent volume:

# Get into the vLLM pod
kubectl exec -it $(kubectl get pods | grep vllm-lora-llama2-7b-deployment-vllm | awk '{print $1}') -- bash

# Inside the pod, download the adapter using Python
mkdir -p /data/lora-adapters
cd /data/lora-adapters
python3 -c "
from huggingface_hub import snapshot_download
adapter_id = 'yard1/llama-2-7b-sql-lora-test'  # Example SQL adapter
sql_lora_path = snapshot_download(
    repo_id=adapter_id,
    local_dir='./sql-lora',
    token=__import__('os').environ['HUGGING_FACE_HUB_TOKEN']
)
"

# Verify the adapter files are downloaded
ls -l /data/lora-adapters/sql-lora

Access the vLLM API

Set up port forwarding to access the vLLM API:

kubectl port-forward svc/vllm-lora-router-service 8000:80

Verify the connection in a new terminal:

curl http://localhost:8000/v1/models

Load and list the models

Forward the port to the vLLM service:

kubectl port-forward svc/vllm-lora-engine-service 8001:80

List available models:

curl http://localhost:8001/v1/models

Load the SQL LoRA adapter:

curl -X POST http://localhost:8001/v1/load_lora_adapter \
    -H "Content-Type: application/json" \
    -d '{
        "lora_name": "sql_adapter",
        "lora_path": "/data/lora-adapters/sql-lora"
    }'

Generate Text with LoRA

Send a query to the vLLM API to generate text using the loaded LoRA adapter:

curl -X POST http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "meta-llama/Llama-2-7b-hf",
        "prompt": "Write a SQL query to select all users who have made a purchase in the last 30 days",
        "max_tokens": 100,
        "temperature": 0.7,
        "lora_adapter": "sql_adapter"
    }'

Unload the adapter:

curl -X POST http://localhost:8001/v1/unload_lora_adapter \
    -H "Content-Type: application/json" \
    -d '{
        "lora_name": "sql_adapter"
    }'