Mathijs Henquet 78203641fe server : Add option to return token pieces in /tokenize endpoint (#9108) ... * server : added with_pieces functionality to /tokenize endpoint * server : Add tokenize with pieces tests to server.feature * Handle case if tokenizer splits along utf8 continuation bytes * Add example of token splitting * Remove trailing ws * Fix trailing ws * Maybe fix ci * maybe this fix windows ci? --------- Co-authored-by: Xuan Son Nguyen <son@huggingface.co>		2024-09-12 22:30:11 +02:00
..
steps.py	server : Add option to return token pieces in /tokenize endpoint (#9108)	2024-09-12 22:30:11 +02:00