diff options
| author | Aleksey Nikiforov <lexn82@gmail.com> | 2025-07-14 12:43:52 -0400 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2025-07-14 18:43:52 +0200 |
| commit | f5353047ef461e6fc9d527e09a06c9802c699929 (patch) | |
| tree | 206c8c56efd3dcac1e39655e73788affe6c02832 /src/unicode.h | |
| parent | 255c22046bcaef41850125be924f3e42e2a65571 (diff) | |
Ported kimi-k2 support from llama.cpp (#609)
Original patch by @gabriellarson:
https://github.com/ggml-org/llama.cpp/pull/14654
Co-authored-by: anikifoss <anikifoss>
Diffstat (limited to 'src/unicode.h')
| -rw-r--r-- | src/unicode.h | 2 |
1 files changed, 2 insertions, 0 deletions
diff --git a/src/unicode.h b/src/unicode.h index 008532a2..48940239 100644 --- a/src/unicode.h +++ b/src/unicode.h @@ -64,4 +64,6 @@ uint8_t unicode_utf8_to_byte(const std::string & utf8); uint32_t unicode_tolower(uint32_t cp); +bool unicode_cpt_is_han(uint32_t cpt); + std::vector<std::string> unicode_regex_split(const std::string & text, const std::vector<std::string> & regex_exprs); |