vllm.distributed.engram_pool.memory
¶
Shard and staging memory: NUMA-local pinned host pages and cuMem HBM.
Classes:
-
CuMem–cuMem HBM: FABRIC-exportable for NVLink peers, or GPUDirect-RDMA capable.
-
SharedHost–Pinned, GPU-mapped host bytes near the current GPU, in a file that RDMA
Functions:
-
device_view–A uint8 tensor over mapped GPU memory; torch retains
ownerwith it.
CuMem
¶
cuMem HBM: FABRIC-exportable for NVLink peers, or GPUDirect-RDMA capable.
Methods:
-
map–Map
handleread-write for this GPU; by default this allocation, at -
release–Drop a handle once mapped: the mapping keeps the memory alive.
Source code in vllm/distributed/engram_pool/memory.py
map(handle=None)
¶
Map handle read-write for this GPU; by default this allocation, at
self.address.
Source code in vllm/distributed/engram_pool/memory.py
SharedHost
¶
Pinned, GPU-mapped host bytes near the current GPU, in a file that RDMA
proxy processes map too: a memfd, or, with thp_dir, an unlinked file on
that tmpfs.
thp_dir must be on a tmpfs mounted with huge=always, so the shard gets
transparent huge pages. A NIC translates every address it reads, and
gathers rows at random: over a large shard on base pages it misses its
translation cache on nearly every row.
Source code in vllm/distributed/engram_pool/memory.py
_check_pages(address, num_bytes, node, huge)
¶
Warn if the pages of [address, address + num_bytes) are mostly off
node, or (huge) mostly not huge pages, as when memory was short or
fragmented. Chunked mbind may have split the range into several VMAs.
Source code in vllm/distributed/engram_pool/memory.py
_claimable_bytes(node)
¶
Free and clean page-cache bytes on NUMA node node.
Source code in vllm/distributed/engram_pool/memory.py
_fault_on_node(owner, node)
¶
Fault in owner's pages on NUMA node node, 1 GB at a time.
MPOL_PREFERRED takes another node's free memory before reclaiming page
cache on node (with the default zone_reclaim_mode=0), so each chunk is
bound to node while it clearly has room, and only preferred after.
Source code in vllm/distributed/engram_pool/memory.py
_huge_page_bytes()
¶
The size of a transparent huge page: 2 MB, or 512 MB on 64 KB pages.
Source code in vllm/distributed/engram_pool/memory.py
device_view(address, num_bytes, owner=None)
¶
A uint8 tensor over mapped GPU memory; torch retains owner with it.