Skip to content

vllm.distributed.engram_pool.proxy

RDMA owner proxy: per GPU, a process whose NIC gathers rows of the shards it serves into other GPUs' landing buffers.

It has its own GIL and one thread that moves rows (its command thread sleeps in recv once set up), so it never waits on a CPU-busy model thread. The GPU process sends commands over a socket (fds by SCM_RIGHTS). For a GPU fetching rows (requester) from GPUs holding them (owners):

  1. The requester's GPU writes, per owner and table, chained RDMA WRITEs whose SGEs gather that owner's rows, a packet's worth each, into the requester's landing buffer, with the owner proxy's pointers. Each QP's last chain links a READ of the requester's memory, which returns only once the rows before it have landed, then a fenced inline WRITE of seq into that QP's flag for the table. The GPU then posts a WRITE_WITH_IMM that copies them into this owner's mailbox (rdma.py).
  2. The owner process, asleep on its receive CQ's completion channel, posts the WRs as they are: its NIC gathers the rows, then writes the flags. Rows in host memory of the requester's node are gathered the same way by the requester's own owner process over loopback QPs, so no GPU reads host memory.

It wakes once per request and never spins.

Run as a script: only the standard library and verbs.py are imported.

Classes:

  • Owner –

    Posts requesters' WRs, so its NIC gathers rows of this GPU's shard and

Functions:

  • chain_length –

    WRs per chain when an owner posts num_wrs WRs over qps QPs.

  • mailbox_bytes –

    A requester's WRs, then their SGEs.

Owner

Posts requesters' WRs, so its NIC gathers rows of this GPU's shard and of its node's host shards; rows and flags of this GPU land through it.

Methods:

  • connect –

    Connect peers' gathering QPs and requesters' request QPs; returns

  • fits –

    Whether the next chain fits its send queue; a large one also needs no

  • map_peer –

    Register another process's host shard (a memfd) for this GPU's

  • register –

    Register host memory or HBM; returns its address, lkey and rkey.

  • retire –

    Reap completed chains, returning their send queue room.

  • run –

    Post requests' WRs a whole chain at a time: small requests (decode)

Source code in vllm/distributed/engram_pool/proxy.py
class Owner:
    """Posts requesters' WRs, so its NIC gathers rows of this GPU's shard and
    of its node's host shards; rows and flags of this GPU land through it."""

    def __init__(self, sock: socket.socket) -> None:
        self.sock = sock
        self.keep: list = []

    def serve(self) -> None:
        while True:
            message, fds, _, _ = socket.recv_fds(self.sock, 1 << 20, 4)
            if not message:
                return
            command, *args = json.loads(message)
            try:
                if command == "register":
                    args.append(fds)
                result = ("ok", getattr(self, command)(*args))
            except Exception:
                result = ("error", traceback.format_exc())
            self.sock.sendall(json.dumps(result).encode())

    def start(self) -> None:
        threading.Thread(target=self.run, daemon=True).start()

    def init(self, device, world, rank, qps_per_peer, sq_depth, max_large):
        self.device = dev = verbs.Device(device)
        # CPUs of another NUMA node post twice as slowly.
        if cpus := verbs.local_cpus(device) & os.sched_getaffinity(0):
            os.sched_setaffinity(0, cpus)
        self.world, self.rank = world, rank
        self.gatherers = [p for p in range(world) if p != rank] + [rank]
        sge_cap = min(dev.max_sge, 16)
        depth = min(sq_depth, dev.max_qp_wr)
        # Gathers complete here unseen until a send queue fills, so it holds
        # every send queue's WRs.
        self.data_cq, _ = dev.create_cq(
            min(dev.max_cqe, len(self.gatherers) * qps_per_peer * depth)
        )
        # Requests complete receives, which wake this process.
        self.recv_cq, self.recv_channel = dev.create_cq(
            len(self.gatherers) * RECV_DEPTH, wakeable=True
        )
        # Per gatherer (this GPU's too, over loopback) QPs for gathers, and per
        # requester one that its requests arrive on.
        self.qps: dict[int, list[int]] = {}
        self.qpns: list[list[int]] = [[] for _ in range(world)]
        notices: list[int] = [0] * world
        self.receivers: dict[int, tuple[int, int]] = {}
        self.by_qpn: dict[int, int] = {}
        for peer in self.gatherers:
            made = [
                dev.create_qp(self.data_cq, self.recv_cq, depth, sge_cap, 64)
                for _ in range(qps_per_peer)
            ]
            self.qps[peer] = [qp for qp, _ in made]
            self.qpns[peer] = [qpn for _, qpn in made]
            self.by_qpn.update((qpn, qp) for qp, qpn in made)
            qp, notices[peer] = dev.create_qp(
                self.data_cq, self.recv_cq, 1, 1, 0, RECV_DEPTH
            )
            self.receivers[notices[peer]] = (peer, qp)
        self.loopback = [
            dev.create_qp(self.data_cq, self.recv_cq, 1) for _ in range(qps_per_peer)
        ]
        self.data_wc = (verbs.ibv_wc * 64)()
        # Per data QP: free send queue slots, and its posted chains' WRs. A
        # QP's last chain's READ and flag stay queued until its next completion.
        self.room = {qp: depth - 2 for qps in self.qps.values() for qp in qps}
        self.large = dict.fromkeys(self.room, 0)
        # Large chains in flight on all QPs, at most max_large (0: no limit).
        self.large_total, self.max_large = 0, max_large or 1 << 30
        self.chains: dict[int, collections.deque] = {
            qp: collections.deque() for qp in self.room
        }
        gid, lid = dev.address()
        return {
            "address": [gid.hex(), lid],
            "qpns": self.qpns,
            "notices": notices,
            "max_sge": sge_cap,
            "mtu": dev.mtu_bytes,
            "device": dev.describe(),
        }

    def connect(self, peers, requesters, box_sizes, max_tables):
        """Connect peers' gathering QPs and requesters' request QPs; returns
        this GPU's mailboxes (per requester and table) and their rkey and lkey."""
        dev = self.device
        for peer, (gid, lid), theirs in peers:
            for qp, remote in zip(self.qps[peer], theirs):
                dev.connect(qp, remote, (bytes.fromhex(gid), lid))
        for qp, qpn, (partner, partner_qpn) in zip(
            self.qps[self.rank], self.qpns[self.rank], self.loopback
        ):
            dev.connect(qp, partner_qpn, dev.address())
            dev.connect(partner, qpn, dev.address())
        self.recv_wr = verbs.ibv_recv_wr()
        by_peer = {peer: qp for peer, qp in self.receivers.values()}
        for peer, (gid, lid), remote in requesters:
            dev.connect(by_peer[peer], remote, (bytes.fromhex(gid), lid))
            for _ in range(RECV_DEPTH):
                dev.post_recv(by_peer[peer], self.recv_wr)
        self.box_sizes = box_sizes
        self.box_starts = [
            sum(box_sizes[:peer]) * max_tables for peer in range(self.world)
        ]
        boxes = mmap.mmap(-1, sum(box_sizes) * max_tables)
        self.keep.append(boxes)
        self.boxes = ctypes.addressof(ctypes.c_char.from_buffer(boxes))
        lkey, rkey = dev.reg_host(self.boxes, len(boxes), verbs.PEER_ORDERED)
        return [self.boxes, rkey, lkey]

    def map_peer(self, pid, fd, size):
        """Register another process's host shard (a memfd) for this GPU's
        loopback gathers; returns its address here and lkey."""
        opened = os.open(f"/proc/{pid}/fd/{fd}", os.O_RDWR)
        mapping = mmap.mmap(opened, size, flags=mmap.MAP_SHARED)
        os.close(opened)
        self.keep.append(mapping)
        address = ctypes.addressof(ctypes.c_char.from_buffer(mapping))
        return address, self.device.reg_host(address, size, verbs.NIC_READ)[0]

    def register(self, kind, size, iova, access, fds):
        """Register host memory or HBM; returns its address, lkey and rkey."""
        (fd,) = fds
        if kind == "host":
            mapping = mmap.mmap(fd, size, flags=mmap.MAP_SHARED)
            os.close(fd)
            address = ctypes.addressof(ctypes.c_char.from_buffer(mapping))
            self.keep.append(mapping)
            return address, *self.device.reg_host(address, size, access)
        keys = self.device.reg_dmabuf(fd, iova, size, access)
        os.close(fd)
        return iova, *keys

    def run(self) -> None:
        """Post requests' WRs a whole chain at a time: small requests (decode)
        first, large ones (prefill) in turn and bounded by `fits`."""
        device, recv_wr = self.device, self.recv_wr
        wc = (verbs.ibv_wc * 64)()
        pending: list[_Request] = []
        device.notify_cq(self.recv_cq)
        while True:
            count = device.poll_cq(self.recv_cq, wc, 64)
            if not count and not pending:
                device.cq_event(self.recv_channel)
                device.notify_cq(self.recv_cq)
                continue
            self.retire()
            _check(wc, count, "request")
            for c in wc[:count]:
                peer, qp = self.receivers[c.qp_num]
                device.post_recv(qp, recv_wr)
                num_wrs, table = c.imm_data & 0xFFFFFF, c.imm_data >> 24
                box = self.boxes + self.box_starts[peer] + table * self.box_sizes[peer]
                pending.append(_Request(self.qps[peer], box, num_wrs))
            # With work pending, poll: its chains complete within microseconds.
            ready = [r for r in pending if self.fits(r)]
            if not ready:
                continue
            # Small requests first; large ones take turns.
            request = next((r for r in ready if not r.large), ready[0])
            for _ in range(CHAINS_PER_TURN):
                if request.posted == request.chains or not self.fits(request):
                    break
                qp, wrs, large = request.next_qp, request.next_wrs, request.large
                device.post_send(qp, request.next_wr)
                self.room[qp] -= wrs
                self.large[qp] += large
                self.large_total += large
                self.chains[qp].append((wrs, large))
                request.posted += 1
            pending.remove(request)
            if request.posted < request.chains:
                pending.append(request)

    def fits(self, request: "_Request") -> bool:
        """Whether the next chain fits its send queue; a large one also needs no
        other large chain on its QP and fewer than `max_large` in flight."""
        qp = request.next_qp
        if self.room[qp] < request.next_wrs:
            return False
        return not request.large or (
            not self.large[qp] and self.large_total < self.max_large
        )

    def retire(self) -> None:
        """Reap completed chains, returning their send queue room."""
        while count := self.device.poll_cq(self.data_cq, self.data_wc, 64):
            _check(self.data_wc, count, "gather")
            # A QP completes its chains in order; each ends in a signaled WR.
            for c in self.data_wc[:count]:
                qp = self.by_qpn[c.qp_num]
                wrs, large = self.chains[qp].popleft()
                self.room[qp] += wrs
                self.large[qp] -= large
                self.large_total -= large

connect(peers, requesters, box_sizes, max_tables)

Connect peers' gathering QPs and requesters' request QPs; returns this GPU's mailboxes (per requester and table) and their rkey and lkey.

Source code in vllm/distributed/engram_pool/proxy.py
def connect(self, peers, requesters, box_sizes, max_tables):
    """Connect peers' gathering QPs and requesters' request QPs; returns
    this GPU's mailboxes (per requester and table) and their rkey and lkey."""
    dev = self.device
    for peer, (gid, lid), theirs in peers:
        for qp, remote in zip(self.qps[peer], theirs):
            dev.connect(qp, remote, (bytes.fromhex(gid), lid))
    for qp, qpn, (partner, partner_qpn) in zip(
        self.qps[self.rank], self.qpns[self.rank], self.loopback
    ):
        dev.connect(qp, partner_qpn, dev.address())
        dev.connect(partner, qpn, dev.address())
    self.recv_wr = verbs.ibv_recv_wr()
    by_peer = {peer: qp for peer, qp in self.receivers.values()}
    for peer, (gid, lid), remote in requesters:
        dev.connect(by_peer[peer], remote, (bytes.fromhex(gid), lid))
        for _ in range(RECV_DEPTH):
            dev.post_recv(by_peer[peer], self.recv_wr)
    self.box_sizes = box_sizes
    self.box_starts = [
        sum(box_sizes[:peer]) * max_tables for peer in range(self.world)
    ]
    boxes = mmap.mmap(-1, sum(box_sizes) * max_tables)
    self.keep.append(boxes)
    self.boxes = ctypes.addressof(ctypes.c_char.from_buffer(boxes))
    lkey, rkey = dev.reg_host(self.boxes, len(boxes), verbs.PEER_ORDERED)
    return [self.boxes, rkey, lkey]

fits(request)

Whether the next chain fits its send queue; a large one also needs no other large chain on its QP and fewer than max_large in flight.

Source code in vllm/distributed/engram_pool/proxy.py
def fits(self, request: "_Request") -> bool:
    """Whether the next chain fits its send queue; a large one also needs no
    other large chain on its QP and fewer than `max_large` in flight."""
    qp = request.next_qp
    if self.room[qp] < request.next_wrs:
        return False
    return not request.large or (
        not self.large[qp] and self.large_total < self.max_large
    )

map_peer(pid, fd, size)

Register another process's host shard (a memfd) for this GPU's loopback gathers; returns its address here and lkey.

Source code in vllm/distributed/engram_pool/proxy.py
def map_peer(self, pid, fd, size):
    """Register another process's host shard (a memfd) for this GPU's
    loopback gathers; returns its address here and lkey."""
    opened = os.open(f"/proc/{pid}/fd/{fd}", os.O_RDWR)
    mapping = mmap.mmap(opened, size, flags=mmap.MAP_SHARED)
    os.close(opened)
    self.keep.append(mapping)
    address = ctypes.addressof(ctypes.c_char.from_buffer(mapping))
    return address, self.device.reg_host(address, size, verbs.NIC_READ)[0]

register(kind, size, iova, access, fds)

Register host memory or HBM; returns its address, lkey and rkey.

Source code in vllm/distributed/engram_pool/proxy.py
def register(self, kind, size, iova, access, fds):
    """Register host memory or HBM; returns its address, lkey and rkey."""
    (fd,) = fds
    if kind == "host":
        mapping = mmap.mmap(fd, size, flags=mmap.MAP_SHARED)
        os.close(fd)
        address = ctypes.addressof(ctypes.c_char.from_buffer(mapping))
        self.keep.append(mapping)
        return address, *self.device.reg_host(address, size, access)
    keys = self.device.reg_dmabuf(fd, iova, size, access)
    os.close(fd)
    return iova, *keys

retire()

Reap completed chains, returning their send queue room.

Source code in vllm/distributed/engram_pool/proxy.py
def retire(self) -> None:
    """Reap completed chains, returning their send queue room."""
    while count := self.device.poll_cq(self.data_cq, self.data_wc, 64):
        _check(self.data_wc, count, "gather")
        # A QP completes its chains in order; each ends in a signaled WR.
        for c in self.data_wc[:count]:
            qp = self.by_qpn[c.qp_num]
            wrs, large = self.chains[qp].popleft()
            self.room[qp] += wrs
            self.large[qp] -= large
            self.large_total -= large

run()

Post requests' WRs a whole chain at a time: small requests (decode) first, large ones (prefill) in turn and bounded by fits.

Source code in vllm/distributed/engram_pool/proxy.py
def run(self) -> None:
    """Post requests' WRs a whole chain at a time: small requests (decode)
    first, large ones (prefill) in turn and bounded by `fits`."""
    device, recv_wr = self.device, self.recv_wr
    wc = (verbs.ibv_wc * 64)()
    pending: list[_Request] = []
    device.notify_cq(self.recv_cq)
    while True:
        count = device.poll_cq(self.recv_cq, wc, 64)
        if not count and not pending:
            device.cq_event(self.recv_channel)
            device.notify_cq(self.recv_cq)
            continue
        self.retire()
        _check(wc, count, "request")
        for c in wc[:count]:
            peer, qp = self.receivers[c.qp_num]
            device.post_recv(qp, recv_wr)
            num_wrs, table = c.imm_data & 0xFFFFFF, c.imm_data >> 24
            box = self.boxes + self.box_starts[peer] + table * self.box_sizes[peer]
            pending.append(_Request(self.qps[peer], box, num_wrs))
        # With work pending, poll: its chains complete within microseconds.
        ready = [r for r in pending if self.fits(r)]
        if not ready:
            continue
        # Small requests first; large ones take turns.
        request = next((r for r in ready if not r.large), ready[0])
        for _ in range(CHAINS_PER_TURN):
            if request.posted == request.chains or not self.fits(request):
                break
            qp, wrs, large = request.next_qp, request.next_wrs, request.large
            device.post_send(qp, request.next_wr)
            self.room[qp] -= wrs
            self.large[qp] += large
            self.large_total += large
            self.chains[qp].append((wrs, large))
            request.posted += 1
        pending.remove(request)
        if request.posted < request.chains:
            pending.append(request)

_Request

A request's WRs in a mailbox, posted a chain at a time over its QPs, as the requester's GPU chained them.

Attributes:

  • next_wrs (int) –

    The next chain's WRs: each QP's last also links a READ and a flag.

Source code in vllm/distributed/engram_pool/proxy.py
class _Request:
    """A request's WRs in a mailbox, posted a chain at a time over its QPs, as
    the requester's GPU chained them."""

    def __init__(self, qps: list[int], box: int, num_wrs: int) -> None:
        self.qps, self.box, self.num_wrs = qps, box, num_wrs
        self.large = num_wrs > SMALL_WRS
        self.chain = chain_length(num_wrs, len(qps))
        self.chains = -(-num_wrs // self.chain)
        self.posted = 0

    @property
    def next_qp(self) -> int:
        return self.qps[self.posted % len(self.qps)]

    @property
    def next_wr(self) -> int:
        return self.box + self.posted * self.chain * WR_BYTES

    @property
    def next_wrs(self) -> int:
        """The next chain's WRs: each QP's last also links a READ and a flag."""
        head = self.posted * self.chain
        flags = 2 if self.posted + len(self.qps) >= self.chains else 0
        return min(self.chain, self.num_wrs - head) + flags

next_wrs property

The next chain's WRs: each QP's last also links a READ and a flag.

_check(wc, count, what)

Raise on the first failed completion in wc.

Source code in vllm/distributed/engram_pool/proxy.py
def _check(wc: ctypes.Array, count: int, what: str) -> None:
    """Raise on the first failed completion in `wc`."""
    for c in wc[:count]:
        if c.status:
            raise RuntimeError(
                f"Engram pool RDMA {what} failed on QP {c.qp_num}: ibv_wc status "
                f"{c.status}, vendor error {c.vendor_err:#x}"
            )

chain_length(num_wrs, qps)

WRs per chain when an owner posts num_wrs WRs over qps QPs.

Source code in vllm/distributed/engram_pool/proxy.py
def chain_length(num_wrs: int, qps: int) -> int:
    """WRs per chain when an owner posts `num_wrs` WRs over `qps` QPs."""
    return min(MAX_CHAIN, max(MIN_CHAIN, -(-num_wrs // qps)))

mailbox_bytes(max_wrs, max_items)

A requester's WRs, then their SGEs.

Source code in vllm/distributed/engram_pool/proxy.py
def mailbox_bytes(max_wrs: int, max_items: int) -> int:
    """A requester's WRs, then their SGEs."""
    return max_wrs * WR_BYTES + max_items * SGE_BYTES