Jared Van Bortel 1b67731e18 BERT tokenizer fixes (#6498) ... Key changes: * BERT conversion: fix abuse of LlamaHfVocab, do not set BOS or EOS * Nomic Embed conversion: pad vocab instead of slicing embedding tensor * llama_tokenize: handle added special tokens like HF does		2024-04-09 13:44:08 -04:00
..
CMakeLists.txt	examples : add tokenize (#4039)	2023-11-17 17:36:44 +02:00
tokenize.cpp	BERT tokenizer fixes (#6498)	2024-04-09 13:44:08 -04:00