tiktoken

Description

tiktoken is a fast BPE tokeniser for use with OpenAI’s models

More information

• Homepage: https://github.com/openai/tiktoken

Availability

Module	Generic	E5	Lake	Epyc	Cascade	Genoa
tiktoken/0.9.0-GCCcore-13.3.0		✓