From ccd6d9cdf6851f7042c48d682daf47bc0e2eca27 Mon Sep 17 00:00:00 2001 From: saood06 Date: Wed, 28 May 2025 00:18:25 -0500 Subject: set cache_prompt default to true (#465) --- examples/server/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'examples/server/README.md') diff --git a/examples/server/README.md b/examples/server/README.md index e17595fe..cb1eb7c9 100644 --- a/examples/server/README.md +++ b/examples/server/README.md @@ -450,7 +450,7 @@ node index.js `id_slot`: Assign the completion task to an specific slot. If is -1 the task will be assigned to a Idle slot. Default: `-1` - `cache_prompt`: Re-use KV cache from a previous request if possible. This way the common prefix does not have to be re-processed, only the suffix that differs between the requests. Because (depending on the backend) the logits are **not** guaranteed to be bit-for-bit identical for different batch sizes (prompt processing vs. token generation) enabling this option can cause nondeterministic results. Default: `false` + `cache_prompt`: Re-use KV cache from a previous request if possible. This way the common prefix does not have to be re-processed, only the suffix that differs between the requests. Because (depending on the backend) the logits are **not** guaranteed to be bit-for-bit identical for different batch sizes (prompt processing vs. token generation) enabling this option can cause nondeterministic results. Default: `true` `system_prompt`: Change the system prompt (initial prompt of all slots), this is useful for chat applications. [See more](#change-system-prompt-on-runtime) -- cgit v1.2.3