FreeBSD.software
Home/textproc/py311-tiktoken

py311-tiktoken

0.12.0_2

Fast BPE tokeniser for use with OpenAI's models

tiktoken is a fast BPE tokeniser for use with OpenAI's models.

Origin: textproc/py-tiktoken
Category: textproc
Size: 2.70MiB
License: MIT
Maintainer: sunpoet@FreeBSD.org
Dependencies: 3 packages
Required by: 7 packages
Website: github.com/openai/tiktoken
$pkg install py311-tiktoken

Dependencies (3)

python311py311-requestspy311-regex

Required By (7 packages)

chatbladepy311-aider_chatpy311-ddtracepy311-langchain-openaipy311-litellmpy311-openai-whisperpy311-transformers

More in textproc

libxml22.15.2
XML parser library for GNOME
expat2.7.4
XML 1.0 parser written in C
qt5-xml5.15.18p109
Qt SAX and DOM implementations (KDE patched)
kf6-kcodecs6.22.0
String encoding library
libxslt1.1.45
XML stylesheet transformation library
rubygem-nokogiri1.19.1
HTML, XML, SAX, and Reader parser
aspell0.60.8.1_1,1
Spelling checker with better suggestion logic than ispell
php84-xml8.4.16
The xml shared extension for php
kf6-sonnet6.22.0
Multi-language spell checker
p5-XML-LibXML2.0210_1,1
Interface to Gnome libxml2 library