From 8a5f8573aefc23282200041abbfa12886083334a Mon Sep 17 00:00:00 2001 From: firecoperana Date: Sun, 8 Jun 2025 11:43:21 +0000 Subject: Rpc improvement (#480) * Add RPC backend in device list to override tensors. * rpc : prevent crashes on invalid input (#9040) Add more checks which prevent RPC server from crashing if invalid input is received from client # Conflicts: # ggml/src/ggml-rpc.cpp * rpc : print error message when failed to connect endpoint (#9042) * Fix RPC error * Add vulkan, sycl to rpc backend * add thread in rpc cpu backend * add cache folder and other improvement in rpc * add header file * support for models with non-512 aligned tensors * rpc : do not wait for response when sending RPC_CMD_SET_TENSOR (#12943) RPC_CMD_SET_TENSOR always returns an empty response and we send this 4 times per token. We can improve TG speed if we don't wait for this empty response. The performance impact of this change depends on the network latency. # Conflicts: # ggml/src/ggml-rpc.cpp * fix(rpc): Improve input validation and error handling (#13069) * fix(rpc): Improve input validation and error handling The `rpc-server` was vulnerable to Denial of Service attacks via several RPC commands (`SET_TENSOR`, `GRAPH_COMPUTE`, etc.). Malformed messages could trigger failed assertions (e.g., invalid `ggml_type`) or out-of-bounds reads/writes leading to `GGML_ABORT` calls, crashing the server process. This PR introduces robust input validation and replaces `abort()` calls with graceful error handling: - **Type Validation:** `deserialize_tensor` now checks if the `tensor->type` is within the valid `GGML_TYPE_COUNT` range *before* calling `ggml_new_tensor_4d`. Returns `nullptr` on invalid type. - **Bounds Checks:** Replaced `GGML_ABORT` in `set_tensor`, `set_tensor_hash`, and `get_tensor` handlers with error logging and returning `false` when data/offset parameters are out of buffer bounds. - **Size Checks:** Added safe arithmetic checks (for overflow) in `graph_compute` when calculating required message sizes based on client-provided `n_nodes` and `n_tensors`. Returns early if the reported sizes conflict with the actual message size or would lead to overflow. - **Error Propagation:** - `create_node` now checks for `nullptr` return values from `deserialize_tensor` and its recursive calls, propagating `nullptr` upwards on failure. Uses `find` instead of `at` for safer map access. - `copy_tensor` now checks for `nullptr` from `deserialize_tensor` and sets the response status to failure if deserialization or bounds checks fail. - `graph_compute` now checks for `nullptr` return from `create_node` and returns failure status correctly. The final return value now reflects the actual computation status. These changes improve the RPC server's resilience against malformed client requests, preventing crashes and ensuring errors are handled more gracefully. Signed-off-by: Ville Vesilehto * refactor(rpc): address pr comments removed comments and unnecessary returns Signed-off-by: Ville Vesilehto * refactor(rpc): ambiguous nullptr from create_node rpc_server::create_node could previously return nullptr if the input ID was 0 (valid) or if an internal error (deserialization, recursion failure) occurred (invalid). This ambiguity made error handling difficult for the caller (`graph_compute`). This commit clarifies the meaning of nullptr: - `graph_compute` now checks if the input 'id' was non-zero when `create_node` returns nullptr, correctly identifying failures versus intentional null links. - `create_node` avoids recursive calls for zero IDs and propagates nullptr unambiguously on failure during recursion. Signed-off-by: Ville Vesilehto * refactor(rpc): initial zero check in create_node The caller (`graph_compute`) already checks `id != 0` when handling a `nullptr` return from `create_node`, correctly distinguishing intentional null links from actual errors. This makes the initial `if (id == 0)` check redundant. Also removes the log message when a tensor ID is not found in the provided map which was added in this branch. Signed-off-by: Ville Vesilehto * fix(rpc): Handle get_alloc_size failure in server Check the return value of `server.get_alloc_size` in the RPC server loop. If the call fails, return early to close the connection. Signed-off-by: Ville Vesilehto * refactor(rpc): input size validation in graph_compute Removes detailed, step-by-step size calculations and overflow checks in favor of simpler direct comparisons, assuming 64-bit overflow is unlikely. Signed-off-by: Ville Vesilehto * refactor(rpc): remove extra status code setting Removes the explicit setting of `response.result = GGML_STATUS_FAILED` when `create_node` returns `nullptr` within `graph_compute`. Primary signal is the `false` return value in case of failure. Signed-off-by: Ville Vesilehto * refactor(rpc): remove redundant check for tensor->type Breaks CI on ubuntu-cpu-make. Tensor type is uint32_t, thus the check is not needed. Signed-off-by: Ville Vesilehto --------- Signed-off-by: Ville Vesilehto # Conflicts: # ggml/src/ggml-rpc.cpp * rpc : fix cache directory initialization (#13188) Signed-off-by: xiaofei # Conflicts: # examples/rpc/rpc-server.cpp * rpc : avoid uninitialized memory in serialize_tensor (#13210) Zero out the name and padding buffers. * fix merge error * Add hello command in RPC * bug fix * add rpc header * fix bug for missing rpc names * add tpc no delay for rpc * add back webui --------- Signed-off-by: Ville Vesilehto Signed-off-by: xiaofei Co-authored-by: firecoperana Co-authored-by: Radoslav Gerganov Co-authored-by: matt23456 Co-authored-by: Ville Vesilehto Co-authored-by: xiaofei Co-authored-by: Justin Santa Barbara --- src/unicode.cpp | 179 ++++++++++++++++++++++++++++++++++---------------------- 1 file changed, 109 insertions(+), 70 deletions(-) (limited to 'src/unicode.cpp') diff --git a/src/unicode.cpp b/src/unicode.cpp index cfffde0d..a57456ea 100644 --- a/src/unicode.cpp +++ b/src/unicode.cpp @@ -18,6 +18,7 @@ #include #include #include +#include size_t unicode_len_utf8(char src) { const size_t lookup[] = { 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 3, 4 }; @@ -25,7 +26,7 @@ size_t unicode_len_utf8(char src) { return lookup[highbits]; } -static std::string unicode_cpts_to_utf8(const std::vector & cps) { +static std::string unicode_cpts_to_utf8(const std::vector& cps) { std::string result; for (size_t i = 0; i < cps.size(); ++i) { result.append(unicode_cpt_to_utf8(cps[i])); @@ -33,7 +34,7 @@ static std::string unicode_cpts_to_utf8(const std::vector & cps) { return result; } -uint32_t unicode_cpt_from_utf8(const std::string & utf8, size_t & offset) { +uint32_t unicode_cpt_from_utf8(const std::string& utf8, size_t& offset) { assert(offset < utf8.size()); if (!(utf8[offset + 0] & 0x80)) { auto result = utf8[offset + 0]; @@ -44,7 +45,7 @@ uint32_t unicode_cpt_from_utf8(const std::string & utf8, size_t & offset) { throw std::invalid_argument("invalid character"); } if (!(utf8[offset + 0] & 0x20)) { - if (offset + 1 >= utf8.size() || ! ((utf8[offset + 1] & 0xc0) == 0x80)) { + if (offset + 1 >= utf8.size() || !((utf8[offset + 1] & 0xc0) == 0x80)) { throw std::invalid_argument("invalid character"); } auto result = ((utf8[offset + 0] & 0x1f) << 6) | (utf8[offset + 1] & 0x3f); @@ -52,7 +53,7 @@ uint32_t unicode_cpt_from_utf8(const std::string & utf8, size_t & offset) { return result; } if (!(utf8[offset + 0] & 0x10)) { - if (offset + 2 >= utf8.size() || ! ((utf8[offset + 1] & 0xc0) == 0x80) || ! ((utf8[offset + 2] & 0xc0) == 0x80)) { + if (offset + 2 >= utf8.size() || !((utf8[offset + 1] & 0xc0) == 0x80) || !((utf8[offset + 2] & 0xc0) == 0x80)) { throw std::invalid_argument("invalid character"); } auto result = ((utf8[offset + 0] & 0x0f) << 12) | ((utf8[offset + 1] & 0x3f) << 6) | (utf8[offset + 2] & 0x3f); @@ -60,7 +61,7 @@ uint32_t unicode_cpt_from_utf8(const std::string & utf8, size_t & offset) { return result; } if (!(utf8[offset + 0] & 0x08)) { - if (offset + 3 >= utf8.size() || ! ((utf8[offset + 1] & 0xc0) == 0x80) || ! ((utf8[offset + 2] & 0xc0) == 0x80) || !((utf8[offset + 3] & 0xc0) == 0x80)) { + if (offset + 3 >= utf8.size() || !((utf8[offset + 1] & 0xc0) == 0x80) || !((utf8[offset + 2] & 0xc0) == 0x80) || !((utf8[offset + 3] & 0xc0) == 0x80)) { throw std::invalid_argument("invalid character"); } auto result = ((utf8[offset + 0] & 0x07) << 18) | ((utf8[offset + 1] & 0x3f) << 12) | ((utf8[offset + 2] & 0x3f) << 6) | (utf8[offset + 3] & 0x3f); @@ -122,10 +123,10 @@ uint32_t unicode_cpt_from_utf8(const std::string & utf8, size_t & offset) { static std::vector unicode_cpt_flags_array() { std::vector cpt_flags(MAX_CODEPOINTS, codepoint_flags::UNDEFINED); - assert (unicode_ranges_flags.front().first == 0); - assert (unicode_ranges_flags.back().first == MAX_CODEPOINTS); + assert(unicode_ranges_flags.front().first == 0); + assert(unicode_ranges_flags.back().first == MAX_CODEPOINTS); for (size_t i = 1; i < unicode_ranges_flags.size(); ++i) { - const auto range_ini = unicode_ranges_flags[i-1]; // codepoint_ini, flags + const auto range_ini = unicode_ranges_flags[i - 1]; // codepoint_ini, flags const auto range_end = unicode_ranges_flags[i]; // codepoint_end, flags for (uint32_t cpt = range_ini.first; cpt < range_end.first; ++cpt) { cpt_flags[cpt] = range_ini.second; @@ -144,7 +145,7 @@ static std::vector unicode_cpt_flags_array() { cpt_flags[p.second].is_uppercase = true; } - for (auto &range : unicode_ranges_nfd) { // start, last, nfd + for (auto& range : unicode_ranges_nfd) { // start, last, nfd cpt_flags[range.nfd].is_nfd = true; } @@ -199,22 +200,55 @@ static std::unordered_map unicode_utf8_to_byte_map() { return map; } -static inline std::wstring unicode_wstring_from_utf8(const std::string & s) { +static inline bool is_valid_utf8(const std::string& str) { + int remaining_bytes = 0; // 当前多字节字符剩余的字节数 + for (unsigned char c : str) { + if (remaining_bytes == 0) { + if ((c & 0x80) == 0x00) continue; // 1字节字符 + else if ((c & 0xE0) == 0xC0) remaining_bytes = 1; // 2字节 + else if ((c & 0xF0) == 0xE0) remaining_bytes = 2; // 3字节 + else if ((c & 0xF8) == 0xF0) remaining_bytes = 3; // 4字节 + else return false; // 非法起始字节 + } + else { + // 检查后续字节是否为10xxxxxx + if ((c & 0xC0) != 0x80) + { + return false; + } + remaining_bytes--; + } + } + return (remaining_bytes == 0); // 确保多字节字符完整 +} + +static inline std::wstring unicode_wstring_from_utf8(const std::string& s) { +#if defined(__clang__) + // disable C++17 deprecation warning for std::codecvt_utf8 +# pragma clang diagnostic push +# pragma clang diagnostic ignored "-Wdeprecated-declarations" +#endif + bool isvalid = is_valid_utf8(s); std::wstring_convert> conv; + +#if defined(__clang__) +# pragma clang diagnostic pop +#endif + return conv.from_bytes(s); } -static std::vector unicode_byte_encoding_process(const std::vector & bpe_words) { +static std::vector unicode_byte_encoding_process(const std::vector& bpe_words) { std::vector bpe_encoded_words; - for (const auto & word : bpe_words) { + for (const auto& word : bpe_words) { std::string text_utf; - auto utf_word = unicode_cpts_from_utf8(word); + auto utf_word = unicode_cpts_from_utf8(word); for (size_t i = 0; i < utf_word.size(); ++i) { text_utf += unicode_cpt_to_utf8(utf_word[i]); } std::string encoded_token; - for (char & c : text_utf) { + for (char& c : text_utf) { encoded_token += unicode_byte_to_utf8(c); } bpe_encoded_words.emplace_back(encoded_token); @@ -223,7 +257,7 @@ static std::vector unicode_byte_encoding_process(const std::vector< } // GPT2 system regex: 's|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+ -static std::vector unicode_regex_split_custom_gpt2(const std::string & text, const std::vector & offsets) { +static std::vector unicode_regex_split_custom_gpt2(const std::string& text, const std::vector& offsets) { std::vector bpe_offsets; // store the offset of each word bpe_offsets.reserve(offsets.size()); // Reserve memory for the approximate size @@ -237,16 +271,16 @@ static std::vector unicode_regex_split_custom_gpt2(const std::string & t start = offset_end; static const uint32_t OUT_OF_RANGE = 0xFFFFFFFF; - auto _get_cpt = [&] (const size_t pos) -> uint32_t { + auto _get_cpt = [&](const size_t pos) -> uint32_t { return (offset_ini <= pos && pos < offset_end) ? cpts[pos] : OUT_OF_RANGE; }; - auto _get_flags = [&] (const size_t pos) -> codepoint_flags { + auto _get_flags = [&](const size_t pos) -> codepoint_flags { return (offset_ini <= pos && pos < offset_end) ? unicode_cpt_flags(cpts[pos]) : codepoint_flags{}; }; size_t _prev_end = offset_ini; - auto _add_token = [&] (const size_t end) -> size_t { + auto _add_token = [&](const size_t end) -> size_t { assert(_prev_end <= end && end <= offset_end); size_t len = end - _prev_end; if (len > 0) { @@ -262,29 +296,29 @@ static std::vector unicode_regex_split_custom_gpt2(const std::string & t return len; }; - for (size_t pos = offset_ini; pos < offset_end; /*pos++*/ ) { + for (size_t pos = offset_ini; pos < offset_end; /*pos++*/) { const uint32_t cpt = _get_cpt(pos); const auto flags = _get_flags(pos); // regex: 's|'t|'re|'ve|'m|'ll|'d - if (cpt == '\'' && pos+1 < offset_end) { - uint32_t cpt_next = _get_cpt(pos+1); + if (cpt == '\'' && pos + 1 < offset_end) { + uint32_t cpt_next = _get_cpt(pos + 1); if (cpt_next == 's' || cpt_next == 't' || cpt_next == 'm' || cpt_next == 'd') { - pos += _add_token(pos+2); + pos += _add_token(pos + 2); continue; } - if (pos+2 < offset_end) { - uint32_t cpt_next_next = _get_cpt(pos+2); + if (pos + 2 < offset_end) { + uint32_t cpt_next_next = _get_cpt(pos + 2); if ((cpt_next == 'r' && cpt_next_next == 'e') || (cpt_next == 'v' && cpt_next_next == 'e') || (cpt_next == 'l' && cpt_next_next == 'l')) { - pos += _add_token(pos+3); + pos += _add_token(pos + 3); continue; } } } - auto flags2 = (cpt == ' ' ? _get_flags(pos+1) : flags); + auto flags2 = (cpt == ' ' ? _get_flags(pos + 1) : flags); // regex: ?\p{L}+ if (flags2.is_letter) { pos += (cpt == ' '); @@ -314,12 +348,12 @@ static std::vector unicode_regex_split_custom_gpt2(const std::string & t } size_t num_whitespaces = 0; - while (_get_flags(pos+num_whitespaces).is_whitespace) { + while (_get_flags(pos + num_whitespaces).is_whitespace) { num_whitespaces++; } // regex: \s+(?!\S) - if (num_whitespaces > 1 && _get_cpt(pos+num_whitespaces) != OUT_OF_RANGE) { + if (num_whitespaces > 1 && _get_cpt(pos + num_whitespaces) != OUT_OF_RANGE) { pos += num_whitespaces - 1; _add_token(pos); continue; @@ -341,7 +375,7 @@ static std::vector unicode_regex_split_custom_gpt2(const std::string & t } // LLAMA3 system regex: "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+" -static std::vector unicode_regex_split_custom_llama3(const std::string & text, const std::vector & offsets) { +static std::vector unicode_regex_split_custom_llama3(const std::string& text, const std::vector& offsets) { std::vector bpe_offsets; // store the offset of each word bpe_offsets.reserve(offsets.size()); // Reserve memory for the approximate size @@ -355,16 +389,16 @@ static std::vector unicode_regex_split_custom_llama3(const std::string & start = offset_end; static const uint32_t OUT_OF_RANGE = 0xFFFFFFFF; - auto _get_cpt = [&] (const size_t pos) -> uint32_t { + auto _get_cpt = [&](const size_t pos) -> uint32_t { return (offset_ini <= pos && pos < offset_end) ? cpts[pos] : OUT_OF_RANGE; }; - auto _get_flags = [&] (const size_t pos) -> codepoint_flags { + auto _get_flags = [&](const size_t pos) -> codepoint_flags { return (offset_ini <= pos && pos < offset_end) ? unicode_cpt_flags(cpts[pos]) : codepoint_flags{}; }; size_t _prev_end = offset_ini; - auto _add_token = [&] (const size_t end) -> size_t { + auto _add_token = [&](const size_t end) -> size_t { assert(_prev_end <= end && end <= offset_end); size_t len = end - _prev_end; if (len > 0) { @@ -380,23 +414,23 @@ static std::vector unicode_regex_split_custom_llama3(const std::string & return len; }; - for (size_t pos = offset_ini; pos < offset_end; /*pos++*/ ) { + for (size_t pos = offset_ini; pos < offset_end; /*pos++*/) { const uint32_t cpt = _get_cpt(pos); const auto flags = _get_flags(pos); // regex: (?i:'s|'t|'re|'ve|'m|'ll|'d) // case insensitive - if (cpt == '\'' && pos+1 < offset_end) { - uint32_t cpt_next = unicode_tolower(_get_cpt(pos+1)); + if (cpt == '\'' && pos + 1 < offset_end) { + uint32_t cpt_next = unicode_tolower(_get_cpt(pos + 1)); if (cpt_next == 's' || cpt_next == 't' || cpt_next == 'm' || cpt_next == 'd') { - pos += _add_token(pos+2); + pos += _add_token(pos + 2); continue; } - if (pos+2 < offset_end) { - uint32_t cpt_next_next = unicode_tolower(_get_cpt(pos+2)); + if (pos + 2 < offset_end) { + uint32_t cpt_next_next = unicode_tolower(_get_cpt(pos + 2)); if ((cpt_next == 'r' && cpt_next_next == 'e') || (cpt_next == 'v' && cpt_next_next == 'e') || (cpt_next == 'l' && cpt_next_next == 'l')) { - pos += _add_token(pos+3); + pos += _add_token(pos + 3); continue; } } @@ -404,7 +438,7 @@ static std::vector unicode_regex_split_custom_llama3(const std::string & // regex: [^\r\n\p{L}\p{N}]?\p{L}+ if (!(cpt == '\r' || cpt == '\n' || flags.is_number)) { - if (flags.is_letter || _get_flags(pos+1).is_letter) { // one or more letters + if (flags.is_letter || _get_flags(pos + 1).is_letter) { // one or more letters pos++; while (_get_flags(pos).is_letter) { pos++; @@ -418,7 +452,7 @@ static std::vector unicode_regex_split_custom_llama3(const std::string & if (flags.is_number) { size_t ini = pos; while (_get_flags(pos).is_number) { - if (++pos - ini >= 3 ) { + if (++pos - ini >= 3) { _add_token(pos); ini = pos; } @@ -428,7 +462,7 @@ static std::vector unicode_regex_split_custom_llama3(const std::string & } // regex: ?[^\s\p{L}\p{N}]+[\r\n]* - auto flags2 = (cpt == ' ' ? _get_flags(pos+1) : flags); + auto flags2 = (cpt == ' ' ? _get_flags(pos + 1) : flags); if (!(flags2.is_whitespace | flags2.is_letter | flags2.is_number) && flags.as_uint()) { pos += (cpt == ' '); while (!(flags2.is_whitespace | flags2.is_letter | flags2.is_number) && flags2.as_uint()) { @@ -444,8 +478,8 @@ static std::vector unicode_regex_split_custom_llama3(const std::string & size_t num_whitespaces = 0; size_t last_end_r_or_n = 0; - while (_get_flags(pos+num_whitespaces).is_whitespace) { - uint32_t cpt2 = _get_cpt(pos+num_whitespaces); + while (_get_flags(pos + num_whitespaces).is_whitespace) { + uint32_t cpt2 = _get_cpt(pos + num_whitespaces); if (cpt2 == '\r' || cpt2 == '\n') { last_end_r_or_n = pos + num_whitespaces + 1; } @@ -460,7 +494,7 @@ static std::vector unicode_regex_split_custom_llama3(const std::string & } // regex: \s+(?!\S) - if (num_whitespaces > 1 && _get_cpt(pos+num_whitespaces) != OUT_OF_RANGE) { + if (num_whitespaces > 1 && _get_cpt(pos + num_whitespaces) != OUT_OF_RANGE) { pos += num_whitespaces - 1; _add_token(pos); continue; @@ -482,7 +516,7 @@ static std::vector unicode_regex_split_custom_llama3(const std::string & } // use std::wregex to split the text -static std::vector unicode_regex_split_stl(const std::wstring & wtext, const std::wstring & regex_expr, const std::vector & offsets) { +static std::vector unicode_regex_split_stl(const std::wstring& wtext, const std::wstring& regex_expr, const std::vector& offsets) { std::wregex expr(regex_expr); std::vector bpe_offsets; // store the offset of each word bpe_offsets.reserve(offsets.size()); // Reserve memory for the approximate size @@ -502,7 +536,7 @@ static std::vector unicode_regex_split_stl(const std::wstring & wtext, c ++it; } - if (start_idx < (int64_t) offset) { + if (start_idx < (int64_t)offset) { bpe_offsets.emplace_back(offset - start_idx); } start += offset; @@ -512,7 +546,7 @@ static std::vector unicode_regex_split_stl(const std::wstring & wtext, c } // use std::regex to split the text -static std::vector unicode_regex_split_stl(const std::string & text, const std::string & regex_expr, const std::vector & offsets) { +static std::vector unicode_regex_split_stl(const std::string& text, const std::string& regex_expr, const std::vector& offsets) { std::regex expr(regex_expr); std::vector bpe_offsets; // store the offset of each word bpe_offsets.reserve(offsets.size()); // Reserve memory for the approximate size @@ -532,7 +566,7 @@ static std::vector unicode_regex_split_stl(const std::string & text, con ++it; } - if (start_idx < (int64_t) offset) { + if (start_idx < (int64_t)offset) { bpe_offsets.emplace_back(offset - start_idx); } start += offset; @@ -541,14 +575,15 @@ static std::vector unicode_regex_split_stl(const std::string & text, con return bpe_offsets; } -static std::vector unicode_regex_split_custom(const std::string & text, const std::string & regex_expr, const std::vector & offsets) { +static std::vector unicode_regex_split_custom(const std::string& text, const std::string& regex_expr, const std::vector& offsets) { std::vector bpe_offsets; if (regex_expr == "'s|'t|'re|'ve|'m|'ll|'d| ?\\p{L}+| ?\\p{N}+| ?[^\\s\\p{L}\\p{N}]+|\\s+(?!\\S)") { bpe_offsets = unicode_regex_split_custom_gpt2(text, offsets); - } else if ( - regex_expr == "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+" || - regex_expr == "(?:'[sS]|'[tT]|'[rR][eE]|'[vV][eE]|'[mM]|'[lL][lL]|'[dD])|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+") { + } + else if ( + regex_expr == "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+" || + regex_expr == "(?:'[sS]|'[tT]|'[rR][eE]|'[vV][eE]|'[mM]|'[lL][lL]|'[dD])|[^\\r\\n\\p{L}\\p{N}]?\\p{L}+|\\p{N}{1,3}| ?[^\\s\\p{L}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+") { bpe_offsets = unicode_regex_split_custom_llama3(text, offsets); } @@ -589,8 +624,8 @@ std::string unicode_cpt_to_utf8(uint32_t cp) { throw std::invalid_argument("invalid codepoint"); } -std::vector unicode_cpts_normalize_nfd(const std::vector & cpts) { - auto comp = [] (const uint32_t cpt, const range_nfd & range) { +std::vector unicode_cpts_normalize_nfd(const std::vector& cpts) { + auto comp = [](const uint32_t cpt, const range_nfd& range) { return cpt < range.first; }; std::vector result(cpts.size()); @@ -602,7 +637,7 @@ std::vector unicode_cpts_normalize_nfd(const std::vector & c return result; } -std::vector unicode_cpts_from_utf8(const std::string & utf8) { +std::vector unicode_cpts_from_utf8(const std::string& utf8) { std::vector result; result.reserve(utf8.size()); size_t offset = 0; @@ -618,7 +653,7 @@ codepoint_flags unicode_cpt_flags(const uint32_t cp) { return cp < cpt_flags.size() ? cpt_flags[cp] : undef; } -codepoint_flags unicode_cpt_flags(const std::string & utf8) { +codepoint_flags unicode_cpt_flags(const std::string& utf8) { static const codepoint_flags undef(codepoint_flags::UNDEFINED); if (utf8.empty()) { return undef; // undefined @@ -632,7 +667,7 @@ std::string unicode_byte_to_utf8(uint8_t byte) { return map.at(byte); } -uint8_t unicode_utf8_to_byte(const std::string & utf8) { +uint8_t unicode_utf8_to_byte(const std::string& utf8) { static std::unordered_map map = unicode_utf8_to_byte_map(); return map.at(utf8); } @@ -642,7 +677,7 @@ uint32_t unicode_tolower(uint32_t cp) { return it == unicode_map_lowercase.end() ? cp : it->second; } -std::vector unicode_regex_split(const std::string & text, const std::vector & regex_exprs) { +std::vector unicode_regex_split(const std::string& text, const std::vector& regex_exprs) { // unicode categories static const std::map k_ucat_enum = { { "\\p{N}", codepoint_flags::NUMBER }, @@ -671,9 +706,9 @@ std::vector unicode_regex_split(const std::string & text, const std // compute collapsed codepoints only if needed by at least one regex bool need_collapse = false; - for (auto & regex_expr : regex_exprs) { + for (auto& regex_expr : regex_exprs) { // search for unicode categories - for (const auto & ucat : k_ucat_enum) { + for (const auto& ucat : k_ucat_enum) { if (std::string::npos != regex_expr.find(ucat.first)) { need_collapse = true; break; @@ -702,18 +737,20 @@ std::vector unicode_regex_split(const std::string & text, const std if (flags.is_whitespace) { //NOTE: C++ std::regex \s does not mach 0x85, Rust and Python regex does. //text_collapsed[i] = (char) 0x85; // as whitespace fallback - text_collapsed[i] = (char) 0x0B; // as whitespace fallback - } else if (k_ucat_cpt.find(flags.category_flag()) != k_ucat_cpt.end()) { + text_collapsed[i] = (char)0x0B; // as whitespace fallback + } + else if (k_ucat_cpt.find(flags.category_flag()) != k_ucat_cpt.end()) { text_collapsed[i] = k_ucat_cpt.at(flags.category_flag()); - } else { - text_collapsed[i] = (char) 0xD0; // fallback + } + else { + text_collapsed[i] = (char)0xD0; // fallback } } } std::vector bpe_offsets = { cpts.size() }; - for (auto & regex_expr : regex_exprs) { + for (auto& regex_expr : regex_exprs) { // first, see if we have an efficient custom regex implementation auto tmp = unicode_regex_split_custom(text, regex_expr, bpe_offsets); @@ -727,7 +764,7 @@ std::vector unicode_regex_split(const std::string & text, const std // if a unicode category is used in the regex, we use the collapsed text and replace the unicode category // with the corresponding collapsed representation bool use_collapsed = false; - for (auto & ucat : k_ucat_enum) { + for (auto& ucat : k_ucat_enum) { if (std::string::npos != regex_expr.find(ucat.first)) { use_collapsed = true; break; @@ -786,7 +823,8 @@ std::vector unicode_regex_split(const std::string & text, const std //printf("text_collapsed: %s\n", text_collapsed.c_str()); //printf("regex_expr_collapsed: %s\n", regex_expr_collapsed.c_str()); bpe_offsets = unicode_regex_split_stl(text_collapsed, regex_expr_collapsed, bpe_offsets); - } else { + } + else { // no unicode category used, we can use std::wregex directly const std::wstring wregex_expr = unicode_wstring_from_utf8(regex_expr); @@ -802,7 +840,8 @@ std::vector unicode_regex_split(const std::string & text, const std //printf("regex_expr: %s\n", regex_expr.c_str()); bpe_offsets = unicode_regex_split_stl(wtext, wregex_expr, bpe_offsets); } - } catch (std::regex_error & e) { + } + catch (std::regex_error& e) { fprintf(stderr, "Failed to process regex: '%s'\n", regex_expr.c_str()); fprintf(stderr, "Regex error: %s\n", e.what()); throw std::runtime_error("Failed to process regex"); @@ -813,7 +852,7 @@ std::vector unicode_regex_split(const std::string & text, const std bpe_words.reserve(bpe_offsets.size()); // reserve memory for the approximate size size_t start = 0; - for (size_t & offset : bpe_offsets) { + for (size_t& offset : bpe_offsets) { bpe_words.emplace_back(); for (size_t i = start; i < start + offset; ++i) { bpe_words.back() += unicode_cpt_to_utf8(cpts[i]); -- cgit v1.2.3