Ka hor intaadan xaraf-taxane u dirin embedding, waxaad qiyaasi kartaa inta token ee uu isticmaali doono adigoo adeegsanaya maktabadda token-qaybiyaha tiktoken ee OpenAI.
Tani si gaar ah ayay waxtar u leedahay, sababtoo ah noocyada embedding-ka (sida text-embedding-3-small) waxay leeyihiin xad token oo ugu sarreeya, kaas oo ay tahay inaadan dhaafin.
---
Sida Token-nada loogu Tiriyo Tiktoken
Waxaad isticmaali kartaa xirmada Python ee tiktoken si aad u xisaabiso tirada token-nada uu xaraf-taxane dhalin doono.
Halkan waxaa ku yaal muunad kooban oo koodh ah:
import tiktoken
def num_tokens_from_string(string: str, encoding_name: str) -> int:
"""Waxay soo celineysaa tirada token ee ku jirta xarig qoraal ah."""
encoding = tiktoken.get_encoding(encoding_name)
num_tokens = len(encoding.encode(string))
return num_tokens
# Tusaale adeegsiga
num_tokens = num_tokens_from_string("tiktoken is great!", "cl100k_base")
print(num_tokens)Muhiim:
Noocyada embedding-ka ee jiilka saddexaad (tusaale, text-embedding-3-small ama text-embedding-3-large), waa inaad isticmaashaa habka codaynta "cl100k_base".
Noocyo kala duwan ayaa u baahan kara habab codayneed oo kala duwan—haddii aadan hubin, mar walba tixraac dukumentiyada nooca.
---
Sababta Tirinta Token-nadu Muhiim u Tahay
Haddii xaraf-taxanahaagu dhaafo cabbirka gelinta ugu badan ee nooca, codsigaaga API-gu wuu fashilmi doonaa.
Tirinta token-nada si sax ah ka hor waxay hubisaa habraacyo embedding oo fudud, waxayna ka hortagtaa khaladaadka inta lagu jiro farsamaynta.
---