#wordcoding — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #wordcoding, aggregated by home.social.
-
To most people the word #token is a black box. I am not using the #tokenizers that are commonly used in #DeepLearning #LLM. Instead I am using my own #WordCoding system that I will call yxxx+. I am using base 16 for coding 300 common ESL English words for my #SLM project. y ranges from 0-F which denotes the #POS (part of speech) of a word. xxx are 3 base 16 digits. Theoretically, I can expand my model to 4000 "base" words. + denotes an additional code which I will explain later. #AI