diff options
| author | Rick G <26732651+TheFlipbook@users.noreply.github.com> | 2024-04-08 06:02:30 -0700 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2024-04-08 16:02:30 +0300 |
| commit | e3c337d87ca650972105a51c6ce302dd236c07ad (patch) | |
| tree | e91b25c531cc5f508e64309071f4a904a0d27189 /common/sampling.h | |
| parent | beea6e1b16e783a0886e78dec01002a8c00db24d (diff) | |
llama : support negative ith in llama_get_ API (#6519)
* llama_sampling_sample with default args is more naively usable
* Batches populated by either llama_batch_get_one or llama_batch_add work with default args
* Previously get_one could use the default argument
* Previously add should usually have used the last index where logits[idx] == true
* This hopefully encourages the use of llama_batch_add
* By giving expected results when using default arguments.
* Adds "negative indexing" feature to llama_get_logits_ith and llama_get_embeddings_ith
* Believed to work with any currently well behaved program
* Default arg now works for both cases (previously would give strange results for add case)
* Any non-negative number is unaffected and behaves as previously
* Negative arguments were previously invalid.
* Implemented as a special case of indexing as suggested by @compilade in https://github.com/ggerganov/llama.cpp/pull/6519
* Fixed mismatch type errors
* cited in macOS CI tests
* Missed in original updates based on PR feedback in https://github.com/ggerganov/llama.cpp/pull/6519
Diffstat (limited to 'common/sampling.h')
| -rw-r--r-- | common/sampling.h | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/common/sampling.h b/common/sampling.h index 56ed991b..639b819a 100644 --- a/common/sampling.h +++ b/common/sampling.h @@ -129,7 +129,7 @@ llama_token llama_sampling_sample( struct llama_sampling_context * ctx_sampling, struct llama_context * ctx_main, struct llama_context * ctx_cfg, - int idx = 0); + int idx = -1); // Prepares and adjusts the set of token candidates for sampling based on penalties, biases, and sampling parameters. llama_token_data_array llama_sampling_prepare( |