diff options
| author | compilade <113953597+compilade@users.noreply.github.com> | 2024-01-22 06:21:52 -0500 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2024-01-22 13:21:52 +0200 |
| commit | d6bd4d46ddb6926087c11e0f6633ab1c81da58c3 (patch) | |
| tree | c75bf322368750ef332c1b981b34f9fd26e6f084 /examples | |
| parent | 152d9d05e097e35f1cac21262946d57faec7542a (diff) | |
llama : support StableLM 2 1.6B (#5052)
* llama : support StableLM 2 1.6B
* convert : fix Qwen's set_vocab wrongly naming all special tokens [PAD{id}]
* convert : refactor Qwen's set_vocab to use it for StableLM 2 too
* nix : add tiktoken to llama-python-extra
* convert : use presence of tokenizer.json to determine StableLM tokenizer loader
It's a less arbitrary heuristic than the vocab size.
Diffstat (limited to 'examples')
0 files changed, 0 insertions, 0 deletions