summaryrefslogtreecommitdiff
path: root/scripts/gen-unicode-data.py
AgeCommit message (Collapse)Author
2024-05-05py : logging and flake8 suppression refactoring (#7081)Brian
Set one as executable and add basicConfig() to another. Also added noqa tag to test scripts.
2024-05-04tests : add test-tokenizer-0.sh + fix some tokenizers (#7036)Georgi Gerganov
* tests : add test-tokenizer-0.sh * unicode : add all unicode number ranges * starcoder : fix pre-tokenizer * tests : add test that fails with DeepSeek tokenizers * falcon : fix regex * unicode : regenerate unicode tables * refact : add tokenizer model * lint : fix * tests : disable failing tests ggml-ci * refact : add tests files ggml-ci * convert : print -> logging ggml-ci * lint : fix * unicode : digit -> number * phi-3 : update