diff options
| author | Georgi Gerganov <ggerganov@gmail.com> | 2023-12-07 13:03:17 +0200 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2023-12-07 13:03:17 +0200 |
| commit | bcc0eb4591bec5ec02fad3f2bdcb1b265052ea56 (patch) | |
| tree | 5082f49b7cb13d8e4f08c14ecf436606a1ae2ff8 /examples/server | |
| parent | 81bc9214a389362010f7a57f4cbc30e5f83a2d28 (diff) | |
llama : per-layer KV cache + quantum K cache (#4309)
* per-layer KV
* remove unnecessary copies
* less code duplication, offload k and v separately
* llama : offload KV cache per-layer
* llama : offload K shift tensors
* llama : offload for rest of the model arches
* llama : enable offload debug temporarily
* llama : keep the KV related layers on the device
* llama : remove mirrors, perform Device -> Host when partial offload
* common : add command-line arg to disable KV cache offloading
* llama : update session save/load
* llama : support quantum K cache (#4312)
* llama : support quantum K cache (wip)
* metal : add F32 -> Q8_0 copy kernel
* cuda : add F32 -> Q8_0 copy kernel
ggml-ci
* cuda : use mmv kernel for quantum cache ops
* llama : pass KV cache type through API
* llama : fix build
ggml-ci
* metal : add F32 -> Q4_0 copy kernel
* metal : add F32 -> Q4_1 copy kernel
* cuda : wip
* cuda : add F32 -> Q4_0 and F32 -> Q4_1 copy kernels
* llama-bench : support type_k/type_v
* metal : use mm kernel only for quantum KV cache
* cuda : add comment
* llama : remove memory_f16 and kv_f16 flags
---------
Co-authored-by: slaren <slarengh@gmail.com>
* readme : add API change notice
---------
Co-authored-by: slaren <slarengh@gmail.com>
Diffstat (limited to 'examples/server')
| -rw-r--r-- | examples/server/server.cpp | 4 |
1 files changed, 0 insertions, 4 deletions
diff --git a/examples/server/server.cpp b/examples/server/server.cpp index 369f81a8..895f751c 100644 --- a/examples/server/server.cpp +++ b/examples/server/server.cpp @@ -2108,10 +2108,6 @@ static void server_params_parse(int argc, char **argv, server_params &sparams, } params.yarn_beta_slow = std::stof(argv[i]); } - else if (arg == "--memory-f32" || arg == "--memory_f32") - { - params.memory_f16 = false; - } else if (arg == "--threads" || arg == "-t") { if (++i >= argc) |