diff options
| author | Pierrick Hymbert <pierrick.hymbert@gmail.com> | 2024-03-09 10:30:04 +0100 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2024-03-09 11:30:04 +0200 |
| commit | fd72d2d2a5e79d61ccef6af3d15f16e5e5cbc352 (patch) | |
| tree | 3a8912adc90c34470fa1e3acb6dd861990159ec7 /examples/server/tests/features/parallel.feature | |
| parent | c2101a2e909ac7c08976d414e64e96c90ee5fa9e (diff) | |
server: tests: add truncated prompt tests, better kv cache size (#5933)
* server: tests: add truncated prompt tests, better size
* server, tests : update regex
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Diffstat (limited to 'examples/server/tests/features/parallel.feature')
| -rw-r--r-- | examples/server/tests/features/parallel.feature | 5 |
1 files changed, 3 insertions, 2 deletions
diff --git a/examples/server/tests/features/parallel.feature b/examples/server/tests/features/parallel.feature index 066698c8..a66fed62 100644 --- a/examples/server/tests/features/parallel.feature +++ b/examples/server/tests/features/parallel.feature @@ -6,8 +6,8 @@ Feature: Parallel Given a server listening on localhost:8080 And a model file tinyllamas/stories260K.gguf from HF repo ggml-org/models And 42 as server seed - And 512 as batch size - And 64 KV cache size + And 128 as batch size + And 256 KV cache size And 2 slots And continuous batching Then the server is starting @@ -76,6 +76,7 @@ Feature: Parallel | disabled | 128 | | enabled | 64 | + Scenario: Multi users with total number of tokens to predict exceeds the KV Cache size #3969 Given a prompt: """ |