Audio Generate API¶

vLLM-Omni provides an API for text-to-audio generation using diffusion-based models such as Stable Audio.

Unlike the Speech API which targets text-to-speech synthesis, the Audio Generate API is designed for general-purpose audio generation from text descriptions (sound effects, music, ambient soundscapes, etc.).

Each server instance runs a single model (specified at startup via vllm-omni serve <model> --omni).

Quick Start¶

Start the Server¶

vllm-omni serve stabilityai/stable-audio-open-1.0 \
    --host 0.0.0.0 \
    --port 8091 \
    --gpu-memory-utilization 0.9 \
    --trust-remote-code \
    --enforce-eager \
    --omni

Generate Audio¶

Using curl:

curl -X POST http://localhost:8091/v1/audio/generate \
    -H "Content-Type: application/json" \
    -d '{
        "input": "The sound of a cat purring",
        "audio_length": 10.0
    }' --output cat.wav

Using Python:

import httpx

response = httpx.post(
    "http://localhost:8091/v1/audio/generate",
    json={
        "input": "The sound of a cat purring",
        "audio_length": 10.0,
    },
    timeout=300.0,
)

with open("cat.wav", "wb") as f:
    f.write(response.content)

API Reference¶

Endpoint¶

POST /v1/audio/generate
Content-Type: application/json

Request Parameters¶

Diffusion Parameters¶

Response Format¶

Returns binary audio data with the appropriate Content-Type header:

Examples¶

Basic Generation¶

Generate audio with only a text prompt (model defaults for all other parameters):

curl -X POST http://localhost:8091/v1/audio/generate \
    -H "Content-Type: application/json" \
    -d '{
        "input": "The sound of ocean waves crashing on a beach"
    }' --output ocean.wav

Custom Duration¶

Specify an explicit audio length in seconds:

curl -X POST http://localhost:8091/v1/audio/generate \
    -H "Content-Type: application/json" \
    -d '{
        "input": "A dog barking",
        "audio_length": 5.0
    }' --output dog_5s.wav

High Quality with Negative Prompt¶

Use a negative prompt to steer generation away from undesired characteristics, and increase inference steps for higher quality:

curl -X POST http://localhost:8091/v1/audio/generate \
    -H "Content-Type: application/json" \
    -d '{
        "input": "A piano playing a gentle melody",
        "audio_length": 10.0,
        "negative_prompt": "Low quality, distorted, noisy",
        "guidance_scale": 8.0,
        "num_inference_steps": 150
    }' --output piano_hq.wav

Reproducible Generation¶

Set a seed to get deterministic results across runs:

curl -X POST http://localhost:8091/v1/audio/generate \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Thunder and rain sounds",
        "audio_length": 15.0,
        "seed": 42
    }' --output thunder.wav

Full Control¶

Combine all parameters for precise control over generation:

curl -X POST http://localhost:8091/v1/audio/generate \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Thunder and rain sounds",
        "audio_length": 15.0,
        "negative_prompt": "Low quality",
        "guidance_scale": 7.0,
        "num_inference_steps": 100,
        "seed": 42
    }' --output thunder_rain.wav

Quick Generation (Fewer Steps)¶

For faster generation with slightly lower quality:

curl -X POST http://localhost:8091/v1/audio/generate \
    -H "Content-Type: application/json" \
    -d '{
        "input": "Birds chirping in a forest",
        "audio_length": 8.0,
        "num_inference_steps": 50
    }' --output birds_quick.wav

Python Client¶

import httpx

response = httpx.post(
    "http://localhost:8091/v1/audio/generate",
    json={
        "input": "Thunder and rain",
        "audio_length": 15.0,
        "negative_prompt": "Low quality",
        "guidance_scale": 7.0,
        "num_inference_steps": 100,
        "seed": 42,
        "response_format": "wav",
    },
    timeout=300.0,
)

with open("thunder.wav", "wb") as f:
    f.write(response.content)

Parameter Tuning Guide¶

guidance_scale¶

Controls how closely the generated audio follows the text prompt.

num_inference_steps¶

Controls the number of denoising steps in the diffusion process.

audio_length¶

Duration of the generated audio clip. For stable-audio-open-1.0, the maximum is approximately 47 seconds. If omitted, the model uses its own default length.

negative_prompt¶

Describes characteristics to avoid. Common negative prompts include:

• "Low quality, distorted, noisy"
• "Silence, static"
• "Music" (when generating sound effects only)

Supported Models¶

Error Responses¶

400 Bad Request¶

Invalid or missing parameters:

{
    "error": {
        "message": "Audio generation model did not produce audio output.",
        "type": "BadRequestError",
        "param": null,
        "code": 400
    }
}

404 Not Found¶

Model mismatch:

{
    "error": {
        "message": "The model `xxx` does not exist.",
        "type": "NotFoundError",
        "param": "model",
        "code": 404
    }
}

422 Unprocessable Entity¶

Pydantic validation failure (e.g. invalid response_format, speed out of range):

{
    "detail": [
        {
            "type": "literal_error",
            "msg": "Input should be 'wav', 'pcm', 'flac', 'mp3', 'aac' or 'opus'",
            ...
        }
    ]
}

Troubleshooting¶

"Audio generation model did not produce audio output"¶

The model finished but returned no audio data. Verify the server started successfully and the model loaded without errors.

Server Not Responding¶

# Check if the server is healthy
curl http://localhost:8091/health

Audio Quality Issues¶

• Increase num_inference_steps (e.g. 150).
• Add a negative prompt: "Low quality, distorted, noisy".
• Increase guidance_scale for stronger prompt adherence.

Generation Timeout¶

• Reduce num_inference_steps.
• Reduce audio_length.
• Check GPU memory with nvidia-smi.

Out of Memory¶

• Lower --gpu-memory-utilization (e.g. 0.8).
• Reduce audio_length.

Development¶

Enable debug logging:

vllm-omni serve stabilityai/stable-audio-open-1.0 \
    --host 0.0.0.0 \
    --port 8091 \
    --gpu-memory-utilization 0.9 \
    --trust-remote-code \
    --enforce-eager \
    --omni \
    --uvicorn-log-level debug