diff options
| author | Jared Van Bortel <jared@nomic.ai> | 2024-04-09 13:44:08 -0400 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2024-04-09 13:44:08 -0400 |
| commit | 1b67731e184e27a465b8c5476061294a4af668ea (patch) | |
| tree | 15a2d877029fb509a34e462c227475bc7d6dc31e /common/common.h | |
| parent | c4a3a4ff47d62d2503ddf9bd91b58c21f04fe3c3 (diff) | |
BERT tokenizer fixes (#6498)
Key changes:
* BERT conversion: fix abuse of LlamaHfVocab, do not set BOS or EOS
* Nomic Embed conversion: pad vocab instead of slicing embedding tensor
* llama_tokenize: handle added special tokens like HF does
Diffstat (limited to 'common/common.h')
| -rw-r--r-- | common/common.h | 8 |
1 files changed, 4 insertions, 4 deletions
diff --git a/common/common.h b/common/common.h index 4635e05d..a7f476c1 100644 --- a/common/common.h +++ b/common/common.h @@ -223,14 +223,14 @@ void llama_batch_add( std::vector<llama_token> llama_tokenize( const struct llama_context * ctx, const std::string & text, - bool add_bos, - bool special = false); + bool add_special, + bool parse_special = false); std::vector<llama_token> llama_tokenize( const struct llama_model * model, const std::string & text, - bool add_bos, - bool special = false); + bool add_special, + bool parse_special = false); // tokenizes a token into a piece // should work similar to Python's `tokenizer.id_to_piece` |