diff options
| author | DAN™ <dranger003@gmail.com> | 2024-05-05 01:19:30 -0400 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2024-05-05 08:19:30 +0300 |
| commit | 889bdd76866ea31a7625ec2dcea63ff469f3e981 (patch) | |
| tree | bcfaea83b66f89181980b64dba57a8c8f8f31a49 /requirements | |
| parent | 6fbd43221167bf96112f899daf22c127b282cbcf (diff) | |
command-r : add BPE pre-tokenization (#7063)
* Add BPE pre-tokenization for Command-R/R+.
* Bump transformers convert requirement.
* command-r : add individual digits regex
---------
Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
Diffstat (limited to 'requirements')
| -rw-r--r-- | requirements/requirements-convert.txt | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/requirements/requirements-convert.txt b/requirements/requirements-convert.txt index a3d6ecec..5520ba73 100644 --- a/requirements/requirements-convert.txt +++ b/requirements/requirements-convert.txt @@ -1,5 +1,5 @@ numpy~=1.24.4 sentencepiece~=0.1.98 -transformers>=4.35.2,<5.0.0 +transformers>=4.40.1,<5.0.0 gguf>=0.1.0 protobuf>=4.21.0,<5.0.0 |