diff options
| author | Elaine <elaine.zosa@gmail.com> | 2024-06-14 13:16:49 +0300 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2024-06-14 13:16:49 +0300 |
| commit | 41b9260f18eb7f325c952006ac46afc1d0d8ad2f (patch) | |
| tree | b5f1efc8d59d9433892cfef23c697dd20e912f6b /convert-hf-to-gguf.py | |
| parent | 172c8256840ffd882ab9992ecedbb587d9b21f15 (diff) | |
convert : add Poro-34B-chat tokenizer support (#7713)
* support for Poro chat pre-tokenizer
* add support for Poro pre-tokenizer
* Update convert-hf-to-gguf-update.py
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* Change Poro-34B-chat to poro-chat
* Change Poro-34B-chat to poro-chat
* Update convert-hf-to-gguf-update.py
* Update llama.cpp
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Diffstat (limited to 'convert-hf-to-gguf.py')
| -rwxr-xr-x | convert-hf-to-gguf.py | 3 |
1 files changed, 3 insertions, 0 deletions
diff --git a/convert-hf-to-gguf.py b/convert-hf-to-gguf.py index 025405a2..55ce502d 100755 --- a/convert-hf-to-gguf.py +++ b/convert-hf-to-gguf.py @@ -477,6 +477,9 @@ class Model: if chkhsh == "c136ed14d01c2745d4f60a9596ae66800e2b61fa45643e72436041855ad4089d": # ref: https://huggingface.co/abacusai/Smaug-Llama-3-70B-Instruct res = "smaug-bpe" + if chkhsh == "c7ea5862a53e4272c035c8238367063e2b270d51faa48c0f09e9d5b54746c360": + # ref: https://huggingface.co/LumiOpen/Poro-34B-chat + res = "poro-chat" if chkhsh == "7967bfa498ade6b757b064f31e964dddbb80f8f9a4d68d4ba7998fcf281c531a": # ref: https://huggingface.co/jinaai/jina-embeddings-v2-base-code res = "jina-v2-code" |