From ff5a3f0c09dfa0a8e0bf76d1748df5c6dee0e8ff Mon Sep 17 00:00:00 2001 From: goerch Date: Tue, 3 Oct 2023 09:16:26 +0200 Subject: Work on the BPE tokenizer (#3252) * Work on the BPE tokenizer Tokenizer tests work for Falcon-7B * Try to fix build problem * Fix debug assertion failure * Fix MSVC Unicode BOM problem * Cleanup and an improvement * Fix compiler warning * Cleanup * Test doesn't work over the full range of Unicodes * Update .gitignore and Makefile * Another Makefile rule * Testing Aquila * Moving byte decoding back to `token_to_piece` ... ... because everyone is using it. * Guarding some unusable code pathes * Streamlining code and adding some more assertions Important change: I'm classifying added tokens as control tokens now for BPE. * Adding a comment * Adding another assertion * Fixed vocabulary guarding assertions * Fix PR for recent change * Fix PR for recent change * Fix for compiler warning * Fix PR for recent change * Fix PR for recent change * Fix PR for recent change * Fix for compiler warning * Fixes for more compiler warnings * Remove unused code * Fix initialization of static maps * Add scores and token types back, adapt gptneox * Update llama.cpp Co-authored-by: Georgi Gerganov * Update unicode.h Co-authored-by: Georgi Gerganov * Update unicode.h Co-authored-by: Georgi Gerganov * Ported Starcoder and added some assertions * Fix coding style * Apply @jploski 's fix for missing tokens --------- Co-authored-by: Georgi Gerganov --- common/common.cpp | 1 + 1 file changed, 1 insertion(+) (limited to 'common') diff --git a/common/common.cpp b/common/common.cpp index 4b233786..7370017f 100644 --- a/common/common.cpp +++ b/common/common.cpp @@ -923,6 +923,7 @@ std::string llama_detokenize_bpe(llama_context * ctx, const std::vector