Sadurunge ngirim string supaya digawe embedding, sampeyan bisa ngira cacah token sing bakal digunakake nganggo pustaka tokenizer tiktoken saka OpenAI.
Iki migunani banget amarga model embedding (kayata text-embedding-3-small) duwe wates maksimal token sing ora kena diluwihi.
---
Carane Ngetung Token nganggo Tiktoken
Sampeyan bisa nggunakake paket Python tiktoken kanggo ngitung jumlah token sing bakal digawe dening string.
Iki conto cuplikan kode:
import tiktoken
def num_tokens_from_string(string: str, encoding_name: str) -> int:
"""Ngasilake jumlah token ing string teks."""
encoding = tiktoken.get_encoding(encoding_name)
num_tokens = len(encoding.encode(string))
return num_tokens
# Conto panggunaan
num_tokens = num_tokens_from_string("tiktoken is great!", "cl100k_base")
print(num_tokens)Penting:
Kanggo model embedding generasi katelu (umpamane, text-embedding-3-small utawa text-embedding-3-large), sampeyan kudu nggunakake enkoding "cl100k_base".
Saben model bisa mbutuhake enkoding sing beda-beda — yen ora yakin, tansah delengen dokumentasi modele.
---
Napa Ngetung Token Penting
Yen string sampeyan ngluwihi ukuran input maksimum model, panjaluk API sampeyan bakal gagal.
Ngitung token kanthi akurat sadurunge bakal njamin alur kerja embedding luwih lancar lan nyegah kesalahan nalika diproses.
---