update README

Author: louie-tsai

tools/recipes/README.md

@@ -2,6 +2,24 @@
 
 Utilities for consuming deployment configurations from [vLLM Recipes](https://recipes.vllm.ai/) and converting them into files that can be used directly with vLLM.
 
+
+## Optimized Deployment Flow
+
+The recipe provides the validated deployment baseline. Hardware and workload
+information are optional inputs that can refine deployment-sensitive values
+before the generated configuration is passed to vLLM.
+
+```mermaid
+flowchart LR
+    R["vLLM Recipe"] --> C["Recipe Converter"]
+    H["Hardware Info (optional)"] --> C
+    W["Workload Info (optional)"] --> C
+    C --> F["config.yml + env.sh"]
+    F --> D["vLLM Docker Image"]
+    D --> E["OpenAI Endpoint"]
+```
+
+
 ## `recipe_json_to_vllm_config.py`
 
 Converts a hardware-specific vLLM Recipes JSON rendering into:
@@ -157,31 +175,64 @@ Single-process renderings such as `single_node_tp` can be converted directly. Mu
 
 ## Optional Runtime Tuning
 
-The Recipes JSON remains the baseline. vLLM Recipes contain the validated model,
-hardware, strategy, environment variables, and serving arguments for a known
-configuration. Runtime tuning is optional and refines only deployment-sensitive
-arguments when more information about the user's actual hardware or workload is
-available.
+The Recipes JSON remains the baseline. vLLM Recipes already provide validated
+model, hardware, strategy, environment variables, and serving arguments.
+Runtime tuning is optional and is intended for parameters whose best value can
+depend on the actual deployment resources or expected request workload.
+
+### Deployment-Time Parameters
+
+The parameters below may already exist in a recipe. They are candidates for
+deployment-time refinement when the recipe value is missing, generic, or based
+on a validation environment that differs from the user's target deployment.
+
+| Runtime parameter | Why it may need deployment-time refinement | Main decision input | Current draft policy |
+| --- | --- | --- | --- |
+| `tensor-parallel-size` | The effective CPU/NUMA topology available to a container or pod can differ from the system used to validate the recipe. | Hardware topology | Use the largest power-of-two TP value that does not exceed the effective NUMA-node count. |
+| `gpu-memory-utilization` | Available memory can differ by machine size, container limits, and other memory use. The vLLM option name is also used by the CPU backend. | Hardware memory + recipe baseline | Compute a conservative safe fraction, reserve 10%, cap at `0.90`, and never increase a smaller recipe value. If the recipe omits it, start from `0.80`. |
+| `max-num-seqs` | The useful scheduler concurrency depends on the number of requests expected to be active at the same time. | Workload concurrency | Set `max-num-seqs` to `--concurrency`. |
+| `max-num-batched-tokens` | The batching budget depends strongly on request input length and concurrency. | Workload token shape + concurrency | Compute `input_tokens * min(concurrency, 8)`, round up to a power of two, and clamp to `2048..32768`. |
+| `data-parallel-size` | The required replica count depends on the requested throughput and the capacity of one replica. | Capacity target | Keep the recipe value today. `--target-qps` is captured for a future capacity-based policy. |
+
+These policies are intentionally isolated in `runtime_tuning.py` so the
+decision rules can evolve as more benchmark data becomes available.
 
-This is useful because some performance parameters may be absent from a recipe,
-may use a general/default value, or may have been selected for the hardware and
-workload used when that recipe was validated. The converter does not replace
-Recipes; it adapts the recipe baseline to the deployment context available when
-`config.yml` is generated.
+### How Runtime Tuning Determines the Values
 
-### Information Used for Runtime Tuning
+The converter can combine the recipe baseline with optional deployment
+information. Different information sources are used for different parameters:
 
-| Information source | How the tool gets it | Examples | Purpose |
+| Information source | How it is obtained | Examples | Parameters it can help determine |
 | --- | --- | --- | --- |
-| vLLM Recipe | Recipes JSON API or direct recipe JSON | model ID, recipe hardware, strategy, existing `argv`, environment variables | Provides the validated baseline and identifies which hardware tuning policy may be used. |
-| Hardware | `--detect-hardware` | effective NUMA nodes, allowed CPUs, physical cores, per-NUMA total/available memory | Refines topology- and memory-dependent settings. |
-| Workload | Optional CLI inputs | `--input-tokens`, `--output-tokens`, `--concurrency` | Refines scheduler concurrency and batching for the expected request shape. |
-| SLO / capacity | Optional CLI inputs | `--ttft-sla-ms`, `--tpot-sla-ms`, `--target-qps` | Supplies constraints for current or future SLA/capacity-aware policies. |
+| vLLM Recipe | Recipes JSON API or direct recipe JSON | model ID, recipe hardware, strategy, existing `argv`, environment variables | Provides the baseline for all parameters and selects the applicable hardware tuning policy. |
+| Hardware (optional) | `--detect-hardware` | effective NUMA nodes, allowed CPUs, physical cores, per-NUMA total/available memory | `tensor-parallel-size`, `gpu-memory-utilization` |
+| Workload (optional) | User CLI inputs | `--input-tokens`, `--output-tokens`, `--concurrency` | `max-num-seqs`, `max-num-batched-tokens`; output length is also available for future KV-cache-aware policies. |
+| SLO / capacity (optional) | User CLI inputs | `--ttft-sla-ms`, `--tpot-sla-ms`, `--target-qps` | Future SLA-aware batching and `data-parallel-size` capacity decisions. |
 
 All additional inputs are optional. If none are supplied, the converter keeps
 the normal Recipes conversion behavior.
 
-Example with hardware and workload information:
+The effective precedence is:
+
+```text
+vLLM defaults
+    -> vLLM Recipes baseline
+        -> hardware refinement (optional)
+            -> workload / SLO refinement (optional)
+                -> config.yml + env.sh
+```
+
+For example, hardware detection can refine TP and memory sizing without any
+workload input:
+
+```bash
+python3 tools/recipes/recipe_json_to_vllm_config.py \
+  --model meta-llama/Llama-3.1-8B-Instruct \
+  --hardware xeon6 \
+  --detect-hardware
+```
+
+Workload information can be added when it is known:
 
 ```bash
 python3 tools/recipes/recipe_json_to_vllm_config.py \
@@ -195,53 +246,21 @@ python3 tools/recipes/recipe_json_to_vllm_config.py \
   --tpot-sla-ms 100
 ```
 
-### Parameters That May Need Deployment-Time Refinement
-
-vLLM Recipes can already contain the parameters below. Runtime tuning is useful
-when a parameter is missing or when the recipe's validated value does not match
-the resources or workload of the target deployment.
-
-| Runtime parameter | Why the recipe alone may not be enough | Additional information used | Current draft policy |
-| --- | --- | --- | --- |
-| `tensor-parallel-size` | The recipe topology may differ from the NUMA resources actually available to the container or pod. | Effective NUMA-node count from `--detect-hardware` | Use the largest power-of-two TP value that does not exceed the effective NUMA-node count. |
-| `gpu-memory-utilization` | Available memory can differ because of machine size, container limits, or other memory use. The vLLM option name is also used by the CPU backend. | Per-NUMA total/available memory plus the recipe value, if present | Compute a conservative safe fraction, reserve 10%, cap at `0.90`, and never increase a smaller recipe value. If the recipe omits it, start from `0.80`. |
-| `max-num-seqs` | The best scheduler concurrency depends on the expected number of concurrent requests. | `--concurrency` | Set `max-num-seqs` to the requested concurrency. |
-| `max-num-batched-tokens` | The useful batching budget depends strongly on request input length and concurrency. | `--input-tokens` and `--concurrency` | Compute `input_tokens * min(concurrency, 8)`, round up to a power of two, and clamp to `2048..32768`. |
-| `data-parallel-size` | The required replica count depends on target throughput and the capacity of one replica. | `--target-qps` plus a future per-replica capacity model | Keep the recipe value today. `target_qps` is recorded for future capacity-based tuning. |
-
 `--output-tokens`, `--ttft-sla-ms`, and `--tpot-sla-ms` are accepted as policy
 inputs, but the current draft does not force them into an arbitrary formula.
 Output length affects KV-cache residency and request lifetime, while TTFT and
 TPOT constrain how aggressive scheduler batching can be. These inputs can be
 used once benchmark-derived or otherwise validated decision rules are available.
 
-The effective precedence is:
-
-```text
-vLLM defaults
-    -> vLLM Recipes baseline
-        -> optional hardware refinement
-            -> optional workload/SLO refinement
-                -> config.yml + env.sh
-```
-
 ### Runtime-Tuning Hardware Scope
 
 Runtime tuning is selected from the resolved recipe JSON's `hardware` field,
 rather than by inspecting which physical devices happen to be present on the
 host. This matters because a GPU server also exposes its host CPU topology.
 
-The current runtime-tuning policy registry contains `xeon6`:
-
-```bash
-python3 tools/recipes/recipe_json_to_vllm_config.py \
-  --model meta-llama/Llama-3.1-8B-Instruct \
-  --hardware xeon6 \
-  --detect-hardware
-```
-
-A tuning request for unregistered recipe hardware, such as `b200`, fails before
-host hardware detection:
+The current runtime-tuning policy registry contains `xeon6`. A tuning request
+for unregistered recipe hardware, such as `b200`, fails before host hardware
+detection:
 
 ```text
 ERROR: Runtime tuning is not supported for recipe hardware 'b200'.
@@ -251,26 +270,6 @@ Currently supported: xeon6.
 Plain recipe conversion remains available for all recipe hardware. The gate only
 applies when optional runtime-tuning inputs are requested.
 
-### Optimized Deployment Flow
-
-The recipe supplies the validated baseline. The converter can optionally combine
-it with the target hardware and workload information, then generate files that
-can be consumed directly by a standard vLLM Docker image.
-
-```mermaid
-flowchart LR
-    R["vLLM Recipe"] --> C["Recipe Converter"]
-    H["Hardware Info"] --> C
-    W["Optional Workload Info"] --> C
-    C --> F["config.yml + env.sh"]
-    F --> D["vLLM Docker Image"]
-    D --> E["OpenAI Endpoint"]
-```
-
-This keeps optimization knowledge in vLLM Recipes and serving in vLLM. The tool
-is the deployment bridge that converts a validated recipe plus optional local
-deployment information into an immediately consumable vLLM configuration.
-
 The implementation remains modular:
 
 - `hardware_detection.py` collects effective CPU/NUMA/memory information.