diff options
| author | Georgi Gerganov <ggerganov@gmail.com> | 2023-11-23 19:07:56 +0200 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2023-11-23 19:07:56 +0200 |
| commit | 6b0a7420d03b9d13cb0e9439a01ce8476d8bf093 (patch) | |
| tree | f184d281cb47e357e4ead4a93a0d1fe504c74bbe /common/common.h | |
| parent | d103d935c0e75769a6a597f7a64cab72c6cc3e79 (diff) | |
llama : KV cache view API + better KV cache management (#4170)
* llama : keep track of used KV cells + better KV cache management
* llama : zero KV cache used upon clear
ggml-ci
* llama : allow exporting a view of the KV cache (#4180)
* Allow exporting a view of the KV cache
* Allow dumping the sequences per cell in common
* Track max contiguous cells value and position as well
* Fix max contiguous empty cells index calculation
Make dump functions deal with lengths or sequences counts > 10 better
* Fix off by one error in dump_kv_cache_view
* Add doc comments for KV cache view functions
Eliminate cell sequence struct; use llama_seq_id directly
Minor cleanups
* common : add -dkvc arg for enabling kv cache dumps
---------
Co-authored-by: Kerfuffle <44031344+KerfuffleV2@users.noreply.github.com>
Diffstat (limited to 'common/common.h')
| -rw-r--r-- | common/common.h | 11 |
1 files changed, 11 insertions, 0 deletions
diff --git a/common/common.h b/common/common.h index 88fa13fc..2f6fe48a 100644 --- a/common/common.h +++ b/common/common.h @@ -122,6 +122,7 @@ struct gpt_params { bool numa = false; // attempt optimizations that help on some NUMA systems bool verbose_prompt = false; // print prompt tokens before generation bool infill = false; // use infill mode + bool dump_kv_cache = false; // dump the KV cache contents for debugging purposes // multimodal models (see examples/llava) std::string mmproj = ""; // path to multimodal projector @@ -218,3 +219,13 @@ std::string get_sortable_timestamp(); void dump_non_result_info_yaml( FILE * stream, const gpt_params & params, const llama_context * lctx, const std::string & timestamp, const std::vector<int> & prompt_tokens, const char * model_desc); + +// +// KV cache utils +// + +// Dump the KV cache view with the number of sequences per cell. +void dump_kv_cache_view(const llama_kv_cache_view & view, int row_size = 80); + +// Dump the KV cache view showing individual sequences in each cell (long output). +void dump_kv_cache_view_seqs(const llama_kv_cache_view & view, int row_size = 40); |