diff options
| author | Marcus Dunn <51931484+MarcusDunn@users.noreply.github.com> | 2023-10-23 12:40:03 -0700 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2023-10-23 22:40:03 +0300 |
| commit | 5be6c803fa5378f62a1590f3ad8c6b64c7c0c2ce (patch) | |
| tree | 190868e0431070686d797c3c2d86da857b8ba55f /common/train.cpp | |
| parent | 6336701c9378c23c85d1c0e464b663ca2bbb8e60 (diff) | |
llama : remove token functions with `context` args in favor of `model` (#3720)
* added `llama_model_token_*` variants to all the `llama_token_*` functions.
* added `LLAMA_API`
* formatting
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* removed old `llama_token` functions
* changed 3 more functions to take in model
- `llama_token_get_text`
- `llama_token_get_score`
- `llama_token_get_type`
* added back docs
* fixed main.cpp
* changed token functions to use new model variants
* changed token functions to use new model variants
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Diffstat (limited to 'common/train.cpp')
| -rw-r--r-- | common/train.cpp | 6 |
1 files changed, 3 insertions, 3 deletions
diff --git a/common/train.cpp b/common/train.cpp index 154ca56e..3cce5da2 100644 --- a/common/train.cpp +++ b/common/train.cpp @@ -236,8 +236,8 @@ int64_t get_example_targets_batch( int64_t used_samples = 0; ggml_set_f32(target_probs, 0.0f); - llama_token bos = llama_token_bos(lctx); - llama_token eos = llama_token_eos(lctx); + llama_token bos = llama_token_bos(llama_get_model(lctx)); + llama_token eos = llama_token_eos(llama_get_model(lctx)); // printf("%s: example_id=%d n_batch=%d n_train_samples=%zu\n", __func__, example_id, n_batch, n_train_samples); for (int k=0; k<n_batch; ++k) { // printf("%s: batch %d\n", __func__, k); @@ -924,7 +924,7 @@ size_t tokenize_file( for (llama_token token=0; token < n_vocab; ++token) { max_token_text_size = std::max( max_token_text_size, - strlen(llama_token_get_text(lctx, token))); + strlen(llama_token_get_text(llama_get_model(lctx), token))); } // upper bound of context byte length. |