Skip to content

Cache management

Cache management defines reusable state, cache identity, validity, lifecycle, and eviction across AR and diffusion execution.

When upstream reports a multimodal encoder-cache size mismatch, both AR and generation schedulers drain that request into a terminal error output and discard its pending prewarm payload. A rejected cache identifier must not leave a request waiting indefinitely or replay the same error on later steps.

When HiSparse defers asynchronous KV loads, a completed nonempty scheduler step advances the upstream fence and drains both deferred frees and loads. Empty steps leave the fence unchanged so a reused block cannot be loaded while an earlier GPU step still reads it.

Diffusion supplies worker attention specs to the native KV planner and uses its default grouping. Its model-config adapter declares that it does not use a registered model's custom SupportsKVCacheGrouping hook.

Candidate invariants

CACHE-INV-001: Cache identity is complete

Rule: A cache key MUST include every input and execution property that can change the reused value.

CACHE-INV-002: Reuse preserves correctness

Rule: A cache optimization MUST provide a disabled path and MUST NOT reuse state after its validity conditions change.

CACHE-INV-003: Ownership has an end

Rule: Request-scoped and model-scoped cache entries MUST have explicit reset, eviction, or teardown behavior.

Safe-change guide

Test hit, miss, invalidation, disabled, concurrent-request, and teardown paths.