vllm.entrypoints.scale_out.token_in_token_out.logprobs_render
¶
Renders /inference/v1/generate sample logprobs from FlatLogprobs
engine rows without a Python object per entry: the same JSON values as
ServingTokens._create_tokens_logprobs + JSONResponse (orjson may write a
float in other notation, e.g. -0.00001 for -1e-05).
Functions:
-
render_json_with_fragments–contentrendered likeJSONResponse, as parts to concatenate, with -
render_tokens_logprobs–See
_render_tokens_logprobs. Renders of more than one block hold
_LeadTable
¶
Cached prefix + str(id) + suffix per token id below max_ids, so a
block of ids maps to its JSON pieces with one fancy index.
Methods:
-
lookup–The pieces of a non-empty array of ids, in its shape.
Source code in vllm/entrypoints/scale_out/token_in_token_out/logprobs_render.py
lookup(ids)
¶
The pieces of a non-empty array of ids, in its shape.
Source code in vllm/entrypoints/scale_out/token_in_token_out/logprobs_render.py
_dumps(content)
¶
Same serialization as starlette.responses.JSONResponse.render.
Source code in vllm/entrypoints/scale_out/token_in_token_out/logprobs_render.py
_rank_pieces(ranks)
¶
,"rank":<rank> with a rank of 0 sent as null, like rank or None.
_render_tokens_logprobs(sampled_token_ids, engine_rows)
async
¶
The GenerateLogProbs JSON of one choice, as parts to concatenate.
Matches the per-entry path, which builds a dict per row (first
occurrence's key order, last occurrence's value; slot 0 has the engine
rank, slot j rank j), looks the sampled entry up in it and lists all
its items as top_logprobs, clamping logprobs to -9999.0 (NaN too) and
sending rank 0 as null. Returns None for rows it cannot map directly
(sampled id not in slot 0, repeated top-k ids, non-finite values).
Source code in vllm/entrypoints/scale_out/token_in_token_out/logprobs_render.py
90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 | |
render_json_with_fragments(content, key, fragments)
¶
content rendered like JSONResponse, as parts to concatenate, with
content["choices"][i][key] (modified in place) replaced by the
pre-rendered parts fragments[i].
Source code in vllm/entrypoints/scale_out/token_in_token_out/logprobs_render.py
render_tokens_logprobs(sampled_token_ids, container, lock=None)
async
¶
See _render_tokens_logprobs. Renders of more than one block hold
lock: on one event loop, interleaving them would only hold more partial
output at once.