diff options
| author | Ting Sun <suntcrick@gmail.com> | 2024-03-28 16:51:06 +0800 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2024-03-28 09:51:06 +0100 |
| commit | cfc4d75df6399b36153ef739f2c1abee4c114bb8 (patch) | |
| tree | 435c4b0bebaba1200af8ab0baf4a69d5d6b9dbe7 /examples | |
| parent | 6902cb7f2e3479f364ee177118200fb7e4e9fc92 (diff) | |
doc: fix outdated default value of batch size (#6336)
* doc: fix outdated default value of batch size
* doc: add doc for ubatch-size
Diffstat (limited to 'examples')
| -rw-r--r-- | examples/main/README.md | 4 |
1 files changed, 3 insertions, 1 deletions
diff --git a/examples/main/README.md b/examples/main/README.md index 9c83fd3b..bb696b56 100644 --- a/examples/main/README.md +++ b/examples/main/README.md @@ -296,7 +296,9 @@ These options help improve the performance and memory usage of the LLaMA models. ### Batch Size -- `-b N, --batch-size N`: Set the batch size for prompt processing (default: 512). This large batch size benefits users who have BLAS installed and enabled it during the build. If you don't have BLAS enabled ("BLAS=0"), you can use a smaller number, such as 8, to see the prompt progress as it's evaluated in some situations. +- `-b N, --batch-size N`: Set the batch size for prompt processing (default: `2048`). This large batch size benefits users who have BLAS installed and enabled it during the build. If you don't have BLAS enabled ("BLAS=0"), you can use a smaller number, such as 8, to see the prompt progress as it's evaluated in some situations. + +- `-ub N`, `--ubatch-size N`: physical maximum batch size. This is for pipeline parallelization. Default: `512`. ### Prompt Caching |