tokenizers

Fast State-of-the-Art Tokenizers optimized for Research and Production

homepage: https://github.com/huggingface/tokenizers

version	toolchain
0.12.1	GCCcore/10.3.0
0.13.3	GCCcore/12.2.0
0.15.2	GCCcore/12.3.0

(quick links: (all) - 0 - a - b - c - d - e - f - g - h - i - j - k - l - m - n - o - p - q - r - s - t - u - v - w - x - y - z)