#language-modeling — Public Fediverse posts

NanoGPT Slowrun: Language Modeling with Limited Data, Infinite Compute

https://qlabs.sh/slowrun

#HackerNews #NanoGPT #Slowrun #LanguageModeling #LimitedData #InfiniteCompute

Tokenization for language modeling: BPE vs. Unigram Language Modeling (2020)

https://ndingwall.github.io/blog/tokenization

#HackerNews #Tokenization #LanguageModeling #BPE #Unigram #NLP

Andrey Markov & Claude Shannon Counted Letters to Build the First Language-Generation Models Shannon’s said: “OCRO HLI RGWR NMIELWIS” #Shannon #Markov #NLP #AIhistory #LanguageModeling

Andrey Markov & Claude Shannon...

Andrey Markov & Claude Shannon Counted Letters to Build the First Language-Generation Models
Shannon’s said: “OCRO HLI RGWR NMIELWIS”
#Shannon #Markov #NLP #AIhistory #LanguageModeling
https://spectrum.ieee.org/andrey-markov-and-claude-shannon-built-the-first-language-generation-models

🎯 #OuteTTS introduces a novel approach to text-to-speech synthesis using pure #languagemodeling
🔧 Built on #LLaMa architecture with just 350M parameters, featuring:

Zero-shot #voicecloning capability
Integration with #WavTokenizer (75 tokens/sec)
Local deployment via #llamacpp
#GGUF format compatibility

🔍 Technical Implementation:

Audio tokenization process
CTC forced alignment
Structured prompt system
Temperature-adjustable outputs

⚠️ Current Limitations:

Limited vocabulary range
String-only input support
Best performance with shorter sentences
Variable temperature sensitivity

https://github.com/edwko/OuteTTS
https://huggingface.co/OuteAI/OuteTTS-0.1-350M

New #languagemodeling #nlp #ai #paper, led by Angelica Chen! We break the steepest MLM training loss drop into *2* phase changes: first in internal grammatical structure, then external capabilities. Big implications for emergence, simplicity bias, and interpretability! https://arxiv.org/abs/2309.07311

#LanguageModeling is trending, to a large extent because of #ChatGPT. But did you know language modeling has been with us for more than a century? And that it was born of the collaboration of a poet and a mathematician?

Our engineer Carsten Schnober tells us more:
https://blog.esciencecenter.nl/language-modeling-the-first-100-years-357556816148