summaryrefslogtreecommitdiff
path: root/models
AgeCommit message (Expand)Author
2024-05-11llama : lookup word in vocab before doing BPE merges (#7193)Haoxiang Fei
2024-05-08llama : add BPE pre-tokenization for Qwen2 (#7114)Ren Xuancheng
2024-05-05command-r : add BPE pre-tokenization (#7063)DAN™
2024-05-04tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)Georgi Gerganov
2024-04-29llama : fix BPE pre-tokenization (#6920)Georgi Gerganov
2023-12-28gpt2 : Add gpt2 architecture integration (#4555)manikbhandari
2023-11-14stablelm : StableLM support (#3586)Galunid
2023-11-02gguf : remove special-case code for GGUFv1 (#3901)Georgi Gerganov
2023-10-24Add more tokenizer tests (#3742)Galunid
2023-10-22Add test for MPT tokenization (#3728)goerch
2023-10-03Work on the BPE tokenizer (#3252)goerch
2023-08-21gguf : new file format with flexible meta data (beta) (#2398)Georgi Gerganov
2023-03-30Make loading weights 10-100x fasterJustine Tunney
2023-03-22Introduce C-style API (#370)Georgi Gerganov
2023-03-21Add tokenizer test + revert to C++11 (#355)Georgi Gerganov
2023-03-14Add section to README on how to run the project on Android (#130)Radoslav Gerganov
2023-03-10Final touchesGeorgi Gerganov
generated by cgit v1.2.3 (git 2.25.1) at 2025-12-25 10:52:50 +0000