Sadurunge ngirim string kanggo embedding, sampeyan bisa ngira-ngira pira token sing bakal digunakake kanthi ngetrapake pustaka tokenizer tiktoken OpenAI.
Iki migunani banget amarga model embedding (kaya text-embedding-3-small) nduweni wates token maksimum sing kudu sampeyan patuhi.
---
Carane Ngetung Token nganggo Tiktoken
Sampeyan bisa nggunakake paket Python tiktoken kanggo ngitung jumlah token sing bakal digawe dening string.
Iki conto cuplikan kode:
import tiktoken
def num_tokens_from_string(string: str, encoding_name: str) -> int:
"""Ngasilake jumlah token ing string teks."""
encoding = tiktoken.get_encoding(encoding_name)
num_tokens = len(encoding.encode(string))
return num_tokens
# Conto panggunaan
num_tokens = num_tokens_from_string("tiktoken is great!", "cl100k_base")
print(num_tokens)Penting:
Kanggo model embedding generasi katelu (contone,
text-embedding-3-smallutawatext-embedding-3-large), sampeyan kudu nggunakake enkoding"cl100k_base".Model sing beda bisa mbutuhake enkoding sing beda — tansah delengen dokumentasi model yen durung yakin.
---
Napa Ngetung Token Penting
Yen string sampeyan ngluwihi ukuran input maksimum model, panjaluk API sampeyan bakal gagal.
Ngitung token kanthi akurat sadurunge bakal njamin alur kerja embedding luwih lancar lan nyegah kesalahan nalika diproses.
---