diff options
| author | Ren Xuancheng <jklj077@users.noreply.github.com> | 2024-05-08 20:06:43 +0800 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2024-05-08 15:06:43 +0300 |
| commit | 229ffff872f8ad0d21c997d18ee7a23692ae60a0 (patch) | |
| tree | b4c2f055395cc0ccbc2b0ade78da27d429cdb2e1 /llama.h | |
| parent | 1fd9c1741d864d01cd7ec6d67227b92d7bfabf22 (diff) | |
llama : add BPE pre-tokenization for Qwen2 (#7114)
* Add BPE pre-tokenization for Qwen2.
* minor : fixes
---------
Co-authored-by: Ren Xuancheng <17811943+jklj077@users.noreply.github.com>
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Diffstat (limited to 'llama.h')
| -rw-r--r-- | llama.h | 5 |
1 files changed, 3 insertions, 2 deletions
@@ -81,8 +81,9 @@ extern "C" { LLAMA_VOCAB_PRE_TYPE_GPT2 = 7, LLAMA_VOCAB_PRE_TYPE_REFACT = 8, LLAMA_VOCAB_PRE_TYPE_COMMAND_R = 9, - LLAMA_VOCAB_PRE_TYPE_OLMO = 10, - LLAMA_VOCAB_PRE_TYPE_DBRX = 11, + LLAMA_VOCAB_PRE_TYPE_QWEN2 = 10, + LLAMA_VOCAB_PRE_TYPE_OLMO = 11, + LLAMA_VOCAB_PRE_TYPE_DBRX = 12, }; // note: these values should be synchronized with ggml_rope |