tokenizers¶
Fast State-of-the-Art Tokenizers optimized for Research and Production
homepage: https://github.com/huggingface/tokenizers
| version | toolchain |
|---|---|
0.12.1 |
GCCcore/10.3.0 |
0.13.3 |
GCCcore/11.3.0 |
0.13.3 |
GCCcore/12.2.0 |
0.15.2 |
GCCcore/12.3.0 |
0.19.1 |
GCCcore/13.2.0 |
(quick links: (all) - 0 - a - b - c - d - e - f - g - h - i - j - k - l - m - n - o - p - q - r - s - t - u - v - w - x - y - z)