summaryrefslogtreecommitdiff
path: root/convert-hf-to-gguf-update.py
AgeCommit message (Expand)Author
2024-05-21tests : test-tokenizer-0.sh print more info (#7402)Georgi Gerganov
2024-05-19Add StableLM2 pre-tokenizer (#7349)Anas Ahouzi
2024-05-17py : convert-hf-to-gguf-update improvements (#7340)Aarni Koskela
2024-05-13llama : rename jina tokenizers to v2 (#7249)Joan Fontanals
2024-05-11convert : skip unaccessible HF repos (#7210)CrispStrobe
2024-05-11llama : add Jina Embeddings architecture (#6826)Joan Fontanals
2024-05-10convert : print "ignore_merges" fieldGeorgi Gerganov
2024-05-09llama3 custom regex split (#6965)jaime-m-p
2024-05-09Add warning if token is invalid (#7173)Galunid
2024-05-08llama : add BPE pre-tokenization for Qwen2 (#7114)Ren Xuancheng
2024-05-08convert : add BPE pre-tokenization for DBRX (#7132)DAN™
2024-05-08py : also print the normalizersGeorgi Gerganov
2024-05-07Fix OLMo HF to GGUF conversion (#6910)nopperl
2024-05-05command-r : add BPE pre-tokenization (#7063)DAN™
2024-05-05py : logging and flake8 suppression refactoring (#7081)Brian
2024-05-04tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)Georgi Gerganov
2024-05-03convert.py : add python logging instead of print() (#6511)Brian
2024-04-30convert : use utf8 encoding (#7000)Georgi Gerganov
2024-04-29llama : fix BPE pre-tokenization (#6920)Georgi Gerganov