Video Depth API¶
POST /v1/videos/depth estimates a temporally consistent depth map for every frame of a video. It returns the depth as a rendered H.264 video and, on request, as raw per-pixel values. It is a vLLM-Omni extension (OpenAI has no depth endpoint) and is available only when the served model declares video depth support; other models answer 400.
Supported models: Video Depth Anything (depth-anything/Video-Depth-Anything-{Small,Base,Large} for relative depth, depth-anything/Metric-Video-Depth-Anything-{Small,Base,Large} for metric depth). See the Video Depth Anything recipe.
Quick Start¶
Get a grayscale depth video (relative models: near is bright; metric models: far is bright):
curl -s http://localhost:8091/v1/videos/depth -F "[email protected]" \
| jq -r .video | base64 -d > walk_depth.mp4
Get a colored video and the raw depth:
import base64
import io
import numpy as np
import requests
with open("walk.mp4", "rb") as f:
response = requests.post(
"http://localhost:8091/v1/videos/depth",
files={"video": f},
data={"colormap": "inferno", "return_depth": "true"},
)
result = response.json()
with open("walk_depth.mp4", "wb") as f:
f.write(base64.b64decode(result["video"]))
depth = np.load(io.BytesIO(base64.b64decode(result["depth"])))["depth"] # [T, H, W] float16
Request¶
Multipart form fields:
| Field | Type | Default | Description |
|---|---|---|---|
video | file | — | Input video upload (required, at most 512 MiB). The endpoint does not fetch remote URLs. |
input_size | int | 518 | Shorter side of the network input, a multiple of 14 from 14 to 1022. Larger values keep finer detail and cost more memory and time. |
max_res | int | 1280 | Frames whose longer side exceeds this are downscaled first; the output has the downscaled size. -1 keeps the input resolution. |
colormap | string | grayscale | Rendering of the depth video: grayscale or inferno. |
return_depth | bool | false | Also return the raw depth as a base64 NPZ archive. |
model | string | served model | Must match the served model when given. |
Every frame of the upload is processed in decode order. A video may have at most 2048 frames (about 68 s at 30 fps), and the output at most 2048 × 1280 × 1280 depth pixels (frames × height × width, after max_res). Every video within the frame limit fits at the default max_res; with max_res=-1, a 1080p video stops at about 1600 frames and a 4K one at about 400. Larger requests answer 400: lower max_res or split the video into clips.
Response¶
{
"created": 1793546102,
"model": "depth-anything/Video-Depth-Anything-Small",
"width": 1280,
"height": 720,
"num_frames": 195,
"fps": 30.0,
"metric": false,
"depth_min": 0.0,
"depth_max": 9.53,
"video": "<base64 MP4>",
"depth": null
}
width,heightare the output size: the input size, or its downscaled size when the longer side exceedsmax_res.metrictells how to read the values. Relative models predict affine-invariant inverse depth (disparity): larger is nearer, and the scale is consistent across the video but has no unit. Metric models predict depth in meters: larger is farther.videois an H.264 MP4 at the input frame rate (24 fps when the container reports none). Values are normalized once over the whole video, fromdepth_mintodepth_max, so brightness is comparable across frames. Videos with an odd width or height gain one black row or column.- With
return_depth=true,depthis a base64 NPZ archive whosedepthentry is a[num_frames, height, width]float16 array.
How long videos are processed¶
The model sees 32 frames at a time. Consecutive windows advance by 22 frames, and each later window also re-reads keyframes of the previous one. For relative models, each window is aligned to the previous one with a least-squares scale and shift over the shared keyframes; for every model, the 8 overlapping frames are cross-faded. Windows are stitched as they finish: every frame outside the 8 still to be cross-faded is final and kept as float16 at once. GPU memory therefore does not grow with the video length, and host memory grows only with the float16 output (about twice its size at the peak, while the frames are gathered into one array).
Offline Inference¶
The same pipeline runs through Omni.generate with the video in multi_modal_data["video"] (a file path, a [T, H, W, 3] uint8 RGB array or a list of PIL images) and the options input_size and max_res in OmniDiffusionSamplingParams.extra_args. The result's multimodal_output["video_depth"] holds depth (a [T, H, W] float16 array), fps, width, height, num_frames and metric.