# Doc writing guide ## Guide to Writing Model Tutorial Doc `docs/source/_templates/Model-Deployment-Tutorial-Template.md` is a template for writing model deployment tutorials. You can copy and modify it to create new docs. ## Testable doc code block generation (``model-code``) - For **documentation authors**: how to insert testable command blocks into docs - For **developers**: how to add a new converter Built-in supported `converter_tag` values: | converter_tag | Renders | YAML source | | --- | --- | --- | | `single_node` | A single node's env exports + `vllm serve` script | `test_cases[case_index]` | | `multi_node` | One host's env exports + `vllm serve` script | `deployment[host_index]` | | `external_dp_template` | One external-DP node's env exports + `vllm serve` command | `templates[host_index]` | | `external_dp_launch` | One `launch_online_dp.py` line per node | `config[]` | | `external_dp_proxy` | The load-balance proxy launch command | `config[]` + `routing` | ### For authors: add a block :::{important} By default, the generator scans only `.md` files under `docs/source/tutorials/models/` and produces artifacts. If you put ``model-code`` blocks in other directories, Sphinx builds will not automatically generate the corresponding scripts. ::: All ``model-code`` blocks need: | Option | Required | Description | | --- | --- | --- | | `block_name` | Yes | Block name; must be unique within the current document | | `converter_tag` | Yes | Selects one of the built-in converters | | `test_case_path` | Yes | Repository-relative YAML path that stays within the repo; the file must exist | Use the body of the block to add shell wrapper lines such as `set -eux`. Always place the `{{ generated }}` placeholder where the converter output should be inserted. #### converter_tag: `single_node` `single_node` reads one item from `test_cases`. The optional `case_index` metadata selects the item; when omitted, it defaults to `0`. Only the fields read by this converter are expanded below. Other test metadata can be left in the YAML and is ignored by this converter. ```yaml test_cases: - name: qwen3-8b-single model: Qwen/Qwen3-8B envs: HCCL_BUFFSIZE: "1024" SERVER_PORT: DEFAULT_PORT server_cmd: - --tensor-parallel-size - "1" - --port - $SERVER_PORT - --trust-remote-code server_cmd_extra: - --enable-expert-parallel benchmarks: ... ``` `envs` is rendered as `export` lines. `SERVER_PORT: DEFAULT_PORT` is resolved to the default single-node port `8000`. `model` becomes `vllm serve `, and `server_cmd` plus optional `server_cmd_extra` become command arguments. Both command fields can be either a shell string or a flat token list. Write the doc block like this: ````md ```{model-code} :block_name: qwen3_8b_single_node :converter_tag: single_node :test_case_path: tests/e2e/nightly/single_node/models/configs/your_model.yaml :case_index: 0 set -eux {{ generated }} ``` ```` Generated shell script: ```bash set -eux export HCCL_BUFFSIZE=1024 export SERVER_PORT=8000 vllm serve Qwen/Qwen3-8B \ --tensor-parallel-size 1 \ --port $SERVER_PORT \ --trust-remote-code \ --enable-expert-parallel ``` #### converter_tag: `multi_node` `multi_node` reads one item from `deployment`. The required `host_index` metadata selects which host to render. ```yaml deployment: - envs: SERVER_PORT: "8000" server_cmd: > vllm serve Qwen/Qwen3-235B-A22B --host 0.0.0.0 --port $SERVER_PORT --data-parallel-size 2 --tensor-parallel-size 8 --data-parallel-address $LOCAL_IP - envs: SERVER_PORT: "8000" server_cmd: > vllm serve Qwen/Qwen3-235B-A22B --headless --port $SERVER_PORT --data-parallel-size 2 --tensor-parallel-size 8 --data-parallel-start-rank 1 --data-parallel-address $MASTER_IP benchmarks: ... ``` `server_cmd` must be a complete command starting with `vllm serve `. It can be written as a shell string or a flat token list. Write the doc block like this: ````md ```{model-code} :block_name: qwen3_235b_worker_1 :converter_tag: multi_node :test_case_path: tests/e2e/nightly/multi_node/internal_dp/config/your_model.yaml :host_index: 1 set -eux {{ generated }} ``` ```` Generated shell script for `host_index: 1`: ```bash set -eux export MASTER_IP=192.168.1.10 export SERVER_PORT=8000 vllm serve Qwen/Qwen3-235B-A22B \ --headless \ --port $SERVER_PORT \ --data-parallel-size 2 \ --tensor-parallel-size 8 \ --data-parallel-start-rank 1 \ --data-parallel-address $MASTER_IP ``` #### converter_tag: `external_dp_template` `external_dp_template` reads one item from `templates`. The required `host_index` metadata selects which template to render. The top-level `model` field is also required because the converter builds `vllm serve `. ```yaml model: Eco-Tech/GLM-Test templates: - node_index: 0 envs: HCCL_BUFFSIZE: "1024" ASCEND_RT_VISIBLE_DEVICES: "${VISIBLE_DEVICES}" server_cmd_template: - --host - 0.0.0.0 - --port - ${PORT} - --data-parallel-size - ${DP_SIZE} - --data-parallel-rank - ${DP_RANK} - --data-parallel-address - ${DP_ADDRESS} - --data-parallel-rpc-port - ${DP_RPC_PORT} - --tensor-parallel-size - ${TP_SIZE} - --trust-remote-code config: ... routing: ... ``` Known braced template variables are rewritten to the positional shell arguments that `run_dp_template.sh` receives from `launch_online_dp.py`: | Template variable | Rendered positional | | --- | --- | | `${VISIBLE_DEVICES}` | `$1` | | `${PORT}` | `$2` | | `${DP_SIZE}` | `$3` | | `${DP_RANK}` | `$4` | | `${DP_ADDRESS}` | `$5` | | `${DP_RPC_PORT}` | `$6` | | `${TP_SIZE}` | `$7` | Unknown braced variables and unbraced shell references such as `$SERVER_PORT` are left unchanged. Write the doc block like this: ````md ```{model-code} :block_name: glm_external_dp_template_node0 :converter_tag: external_dp_template :test_case_path: tests/e2e/nightly/multi_node/external_dp/config/your_model.yaml :host_index: 0 set -eux export HCCL_IF_IP=$local_ip export GLOO_SOCKET_IFNAME=$nic_name export TP_SOCKET_IFNAME=$nic_name export HCCL_SOCKET_IFNAME=$nic_name {{ generated }} ``` ```` Generated shell script for `host_index: 0`: ```bash set -eux export HCCL_IF_IP=$local_ip export GLOO_SOCKET_IFNAME=$nic_name export TP_SOCKET_IFNAME=$nic_name export HCCL_SOCKET_IFNAME=$nic_name export HCCL_BUFFSIZE=1024 export ASCEND_RT_VISIBLE_DEVICES=$1 vllm serve Eco-Tech/GLM-Test \ --host 0.0.0.0 \ --port $2 \ --data-parallel-size $3 \ --data-parallel-rank $4 \ --data-parallel-address $5 \ --data-parallel-rpc-port $6 \ --tensor-parallel-size $7 \ --trust-remote-code ``` #### converter_tag: `external_dp_launch` `external_dp_launch` reads the full `config` list and renders one `launch_online_dp.py` command per node. It does not take an index option. ```yaml config: - node_index: 0 port_start: 7100 dp_rpc_port: 12321 dp_size: 2 dp_size_local: 2 dp_rank_start: 0 tp_size: 8 dp_address: "${NODE_0_IP}" - node_index: 1 port_start: 7200 dp_rpc_port: 12321 dp_size: 4 dp_size_local: 4 dp_rank_start: 0 tp_size: 4 dp_address: "${NODE_1_IP}" templates: ... routing: ... ``` Write the doc block like this: ````md ```{model-code} :block_name: glm_external_dp_launch :converter_tag: external_dp_launch :test_case_path: tests/e2e/nightly/multi_node/external_dp/config/your_model.yaml set -eux {{ generated }} ``` ```` Generated shell script: ```bash set -eux python launch_online_dp.py --dp-size 2 --tp-size 8 --dp-size-local 2 --dp-rank-start 0 --dp-address ${NODE_0_IP} --dp-rpc-port 12321 --vllm-start-port 7100 python launch_online_dp.py --dp-size 4 --tp-size 4 --dp-size-local 4 --dp-rank-start 0 --dp-address ${NODE_1_IP} --dp-rpc-port 12321 --vllm-start-port 7200 ``` #### converter_tag: `external_dp_proxy` `external_dp_proxy` reads `config` and `routing`. It renders the `load_balance_proxy_server_example.py` command for `routing.type: disaggregated_prefill`. It does not take an index option. ```yaml routing: type: disaggregated_prefill groups: prefiller: [0] decoder: [1] config: - node_index: 0 port_start: 7100 dp_size_local: 2 dp_rpc_port: 12321 dp_size: 2 dp_rank_start: 0 tp_size: 8 dp_address: "${NODE_0_IP}" - node_index: 1 port_start: 7200 dp_size_local: 4 dp_rpc_port: 12321 dp_size: 4 dp_rank_start: 0 tp_size: 4 dp_address: "${NODE_1_IP}" templates: ... ``` `routing.groups.prefiller` and `routing.groups.decoder` contain indices into `config`. Each referenced node expands to `dp_size_local` host and port entries. The proxy itself is rendered on `${NODE_0_IP}:1999`. Write the doc block like this: ````md ```{model-code} :block_name: glm_external_dp_proxy :converter_tag: external_dp_proxy :test_case_path: tests/e2e/nightly/multi_node/external_dp/config/your_model.yaml set -eux {{ generated }} ``` ```` Generated shell script: ```bash set -eux python load_balance_proxy_server_example.py \ --host ${NODE_0_IP} \ --port 1999 \ --prefiller-hosts \ ${NODE_0_IP} \ ${NODE_0_IP} \ --prefiller-ports \ 7100 \ 7101 \ --decoder-hosts \ ${NODE_1_IP} \ ${NODE_1_IP} \ ${NODE_1_IP} \ ${NODE_1_IP} \ --decoder-ports \ 7200 \ 7201 \ 7202 \ 7203 ``` ### Local debugging and generation #### Generate only (without building the full site) ```bash # Generate all model-code artifacts under docs/source/tutorials/models/ python3 tools/docs_codegen/cli.py # Generate artifacts for a single document python3 tools/docs_codegen/cli.py --doc docs/source/tutorials/models/Kimi-K2-Thinking.md # Generate a single block and print it (no files written) python3 tools/docs_codegen/cli.py \ --block docs/source/tutorials/models/Kimi-K2-Thinking.md::kimi_k2_thinking_single_node \ --dry-run --stdout ``` By default, artifacts are written to: `docs/_build/doc_codegen//.sh`. :::{note} After the script is generated, please make sure to check whether the generated content is runnable, especially key parts such as environment variables and command-line parameters. ::: #### Build the site & preview locally ```bash # Install documentation build dependencies python3 -m pip install -r docs/requirements-docs.txt # (Optional) Clean previous builds make -C docs clean # Build the English site make -C docs html # (Optional) Build the Chinese site make -C docs intl # Preview locally python3 -m http.server -d docs/_build/html 8000 # Then open in a browser: # http://localhost:8000 ``` ### For developers: add a new converter A converter turns one loaded YAML file plus one parsed `ModelCodeBlock` into a `GeneratedScript`. The current pipeline is: 1. `BlockScanner` parses ``model-code`` fences and accepts only options listed in `MODEL_CODE_OPTION_NAMES`. 2. `YamlLoader` loads `test_case_path`. 3. `get_converter()` looks up `block.converter_tag` from `build_default_converters()`. 4. The selected converter returns `GeneratedScript(content=..., language="shell")`. 5. `GeneratorService` replaces `{{ generated }}` in the block body, validates that the final script is non-empty, and writes `docs/_build/doc_codegen//.sh`. To add a converter: 1. In `tools/docs_codegen/converters.py`, add a `BaseConverter` subclass with a unique `name`. That name is the value authors put in `:converter_tag:`. 2. Implement `convert(self, loaded_yaml, *, block) -> GeneratedScript`. Use `make_docs_codegen_error(..., block=block)` for user-facing validation errors so the CLI and Sphinx output include document context. 3. Reuse helpers from `tools/docs_codegen/utils.py`, such as `require_mapping`, `require_mapping_list`, `require_scalar_mapping`, `require_indexed_mapping`, `require_node_field`, `parse_command_tokens`, `substitute_template_positionals`, and `render_cli_command`. 4. Register the converter in `build_default_converters()`. If it is not registered, `get_converter()` will reject the new `converter_tag`. 5. If the converter needs new directive metadata, add the option name to `MODEL_CODE_OPTION_NAMES` in `tools/docs_codegen/scanner.py` and to `ModelCodeDirective.option_spec` in `tools/docs_codegen/sphinx_extension.py`. Read the option with `block.get_option("")`. 6. Add or update tests in `tests/ut/tools/test_docs_codegen.py`. Cover the successful render path, required option validation, YAML shape validation, and any CLI/Sphinx scanner behavior affected by new metadata. 7. Add a real ``model-code`` example in a model tutorial, preferably under `docs/source/tutorials/models/`, and point it to an existing YAML file under `tests/`. 8. Validate with the CLI: ```bash python3 tools/docs_codegen/cli.py --doc --dry-run python3 tools/docs_codegen/cli.py --block :: --dry-run --stdout ``` If a converter should render something other than shell, set `GeneratedScript.language` accordingly so Sphinx can highlight the generated literal block correctly.