tokenizers

Description

Fast State-of-the-Art Tokenizers optimized for Research and Production

More information

• Homepage: https://github.com/huggingface/tokenizers

Availability

Module	NIC5	Hercules2	Dragon2	Lemaitre3	Dragon1
tokenizers/0.13.3-GCCcore-12.2.0	✓