Ka hor intaadan xaraf-taxane u dirin embedding, waxaad qiyaasi kartaa inta token ee uu isticmaali doono adigoo adeegsanaya maktabadda tokenizer-ka tiktoken ee OpenAI.
Tani si gaar ah ayay waxtar u leedahay, sababtoo ah noocyada embedding-ka (sida text-embedding-3-small) waxay leeyihiin xaddidyo token oo ugu badan oo aad u baahan tahay inaad ku ekaato.
---
Sida Token-nada loogu Tiriyo Tiktoken
Waxaad isticmaali kartaa xirmada Python ee tiktoken si aad u xisaabiso tirada token-nada uu xaraf-taxane dhalin doono.
Halkan waxaa ku yaal muunad kooban oo koodh ah:
import tiktoken
def num_tokens_from_string(string: str, encoding_name: str) -> int:
"""Waxay soo celineysaa tirada token ee ku jirta xarig qoraal ah."""
encoding = tiktoken.get_encoding(encoding_name)
num_tokens = len(encoding.encode(string))
return num_tokens
# Tusaale adeegsiga
num_tokens = num_tokens_from_string("tiktoken is great!", "cl100k_base")
print(num_tokens)Muhiim:
Loogu talagalay noocyada embedding-ka jiilka saddexaad (tusaale,
text-embedding-3-smallamatext-embedding-3-large), waa inaad isticmaashaa koodhaynta"cl100k_base".Noocyo kala duwan waxay u baahan karaan koodhaynno kala duwan — had iyo jeer eeg dukumentiyada nooca haddii aadan hubin.
---
Sababta Tirinta Token-nadu Muhiim u Tahay
Haddii xaraf-taxanahaagu dhaafo cabbirka gelinta ugu badan ee nooca, codsigaaga API-gu wuu fashilmi doonaa.
Tirinta token-nada si sax ah ka hor waxay hubisaa habraacyo embedding oo fudud, waxayna ka hortagtaa khaladaadka inta lagu jiro farsamaynta.
---