Skip to content

vllm_omni.diffusion.models.sam3

Modules:

Name Description
pipeline_sam3

SAM3 promptable concept segmentation pipeline.
sam3_model

SAM3 promptable concept segmentation detector (image path).
sam3_tracker

SAM3 memory tracker for video concept segmentation.
sam3_video

SAM3 video concept segmentation: detect every frame, track with memory, associate.

Sam3Pipeline

Bases: Module, SupportImageInput

Registered for both Sam3Model (image) and Sam3VideoModel (combined) checkpoints.

config instance-attribute 

config = Sam3DetectorConfig.from_dict(config)

detector instance-attribute 

detector = Sam3Detector(self.config)

image_size instance-attribute 

image_size = (
    self.config.vision_config.backbone_config.image_size
)

od_config instance-attribute 

od_config = od_config

supports_video property 

supports_video: bool

tokenizer instance-attribute 

tokenizer = AutoTokenizer.from_pretrained(
    model, revision=od_config.revision
)

tracker instance-attribute 

tracker: Sam3Tracker | None = None

tracker_neck instance-attribute 

tracker_neck: Sam3VisionNeck | None = None

video_config instance-attribute 

video_config: Sam3VideoConfig | None = None

weights_sources instance-attribute 

weights_sources = [
    DiffusersPipelineLoader.ComponentSource(
        model_or_path=model,
        subfolder=None,
        revision=od_config.revision,
        prefix="",
        fall_back_to_pt=False,
        allow_patterns_overrides=[
            "model.safetensors",
            "model-*.safetensors",
            "sam3*.safetensors",
        ],
    )
]

forward 

forward(
    req: DiffusionRequestBatch, **kwargs
) -> DiffusionOutput

load_weights 

load_weights(
    weights: Iterable[tuple[str, Tensor]],
) -> set[str]

segment 

segment(
    image: Image,
    text: str,
    boxes: list[list[float]],
    box_labels: list[int],
    score_threshold: float,
    mask_threshold: float,
) -> dict[str, Any]

segment_video 

segment_video(
    video: Any, prompts: list[str]
) -> dict[str, Any]