#tinygrad — Public Fediverse posts

RT @__tinygrad__: tinygrad wird das C für dich schreiben. Unser neuer Treiber kompiliert alle Interaktionen mit der GPU in C, sodass die CPU im laufenden Betrieb fast nichts mehr tut. Elon Musk (@elonmusk) SpaceX hat fast die V1.0 eines in-house entwickelten AI-Trainingsstapels in C fertiggestellt, der exakt auf 220k GB300s mit 800G-NICs abgebildet ist und stark von Pipeline-Parallelismus Gebrauch macht, um so nah wie möglich an die Bare-Metal-Leistung heranzukommen. Der potenzielle Geschwindigkeitsvorteil gegenüber JAX bei großen Trainingsläufen liegt über einer Größenordnung. — https://nitter.net/elonmusk/status/2059884150187053488#m

mehr auf Arint.info

#AI #CProgramming #GPU #SpaceX #tinygrad #arint_info

https://x.com/__tinygrad__/status/2060064595465912434#m

High speed graphics rendering research with tinygrad/tinyJIT

https://github.com/quantbagel/gtinygrad

#HackerNews #HighSpeedGraphics #RenderingResearch #tinygrad #tinyJIT #GraphicsProgramming #TechInnovation

At long last, the blog post I've been working on for what seems like forever is finished!

https://cprimozic.net/blog/growing-sparse-computational-graphs-with-rnns/

It's packed with lots of really cool stuff: ML #interpretability, #grokking, #tinygrad, #graphviz, and more

Did some #machinelearning
#benchmarks on my 7900 XTX GPU and wrote up notes: https://cprimozic.net/notes/posts/machine-learning-benchmarks-on-the-7900-xtx/

TL;DR performance on AMD GPUs is pretty bad, but it's likely a software issue.

I tested the raw hardware using some scripts from #tinygrad, and the raw FLOPS are excellent.

Speaking of #tinygrad, I've been hanging out in that Discord server for the past month or so

Very unique project. The creator and maintainer geohot is an interesting person for sure.

Lots of good ideas, and the only place I've actually seen all the way from the top to the bottom of a GPU-powered ML stack before

Not a good place to go to learn though; very much a do-it-yourself vibe and noobs aren't tolerated at all

I've been making more progress on the sparse RNN training and visualization

Working on the blog post now. Lots of cool stuff went into this from custom activation functions, custom regularizers, the new machine learning library #tinygrad, #graphviz, #webgl, and more

Here, it learned a gated 3-state state machine coupled with other neurons that perform a different boolean operation depending on the current state

#rnn #machinelearning

Looking forward to seeing what they do

the tiny corp raised $5.1M

https://geohot.github.io/blog/jekyll/update/2023/05/24/the-tiny-corp-raised-5M.html

#geohotz #CommaAI #tinygrad #tinycorp