#puppyslugs — Public Fediverse posts

#LLMs are an amazing #AI experiment with incredibly important research results, but the technology is far too immature to be allowed into production. Nobody should trust its output. We need more research and less development, we need to train LLMs under controlled conditions where we dissect every step in the training process and look at how it changes the patterns inside the artificial neural network.
Ever since #DeepDream and its #puppyslugs, the first AI generated images that got viral on the Internet, researchers have been tinkering with already existing arltificial neural networks to see what a single layer of neurons or even a single neuron does within the system, and what kinds of patterns occur within such a network when it does something, but that's not enough. Instead of making the models bigger and bigger, we need to train much smaller models much faster and much more often, comparing individual outcomes to one another. Since we are dealing with complex systems very much capable of chaotic behaviour, we must look at them not in the way we look at conventional engineering. The AI machinery may be entirely deterministic as far as the mathematics go, using pseudorandom numbers as the noise it needs in order to work, but even if we can reproduce the output by using the same random seed, we still cannot understand how it came to be. All we are doing at the moment is basically just Linear Algebra, but with gazillion dimensional tensors. Even if we trace the path each single bit of input signal takes through the model, we still don't understand how the model makes the output from the input, and that's because we aren't paying enough attention to how exactly the training actually works.

We've got all those cute but unreliable toys now. They may not be fit for production in most cases, but as soon as the LLM cult collapses and the AI bubble bursts, people will find out that much smaller models trained on manually curated data can actually be very useful for all kinds of specialised systems, even though we won't get any closer to #AGI. I think with our current hardware technology we won't even get close to actual human intelligence before the decline of the Industrial Age erodes our global industrial productive capacity to the point where computers become very rare and very expensive again. Our current digital computers are far too energy hungry and far too precise to run anything as complex and noise resistant as a human brain, we'd have to build something analogue and low-power. Something that doesn't compute with discrete numbers but with something like voltage or brightness that can take any value in between 0 and 1. Up to now, we haven't really tried to make analogue signal processing circuits really tiny because we have been using DSPs instead, but what if we tried to make very densely packed silicon chips out of them that mimic the signal pathway topology of a slice of brain? I'm pretty sure there are already people working on that somewhere, but with a pityful budget because all the "AI" funds go to bloody useless LLM chatbots.

When the #AIBubble bursts, there won't be much funding for AI research, but at least more of it will go to fields where actual progress can be made instead of putting it all into #MachineLearning. Machine learning is great, we have made some real progress in the last 20 years because of all the Internet data on which we could train our models, and also because of relatively cheap GPUs to do the heavy lifting, but now GPUs are expensive because of chatbot breeders and cryptobros, and the data on the Internet is far more AI output than anything else, and since AI can't tell AI and humans apart yet (if ever), we are at the point where there won't be any progress in machine learning without a lot of human labour. Even if some mousepad proles from Africa or Asia or Latin America do all the click work, write all the detailed descriptions for visual media and audio, pick out all the useless AI hallucinations that slipped into the proposed training data, it will make the process really, really expensive because this is something that takes a lot of time and can't be automated. So any completely new large scale machine learning models may be a thing of the past soon. Already existing models can be used, and with LoRAs, they can be taught some new tricks, but if we use whatever hype is left to learn as much as we can about the training process and what structures it builds inside the model, we will be able to build better models that can do more with less.

@kriegundfreitag Ich weiß gar nicht, was alle haben - ich pixle schon digitale Bilder seit 1988, und seit es neuronale Netze gibt, die interessante Bilder produzieren (also seit #deepdream mit den #puppyslugs und komischen LSD-Hallus), benutze ich auch die. #txt2img ist ein interessantes Werkzeug, das sich auch gut mit anderen digitalen Techniken kombinieren läßt. Manchmal male und zeichne ich auch mit Acryl oder Bleistift, aber am Ende digitalisiere ich das meist und bearbeite es mit #GIMP und #Krita weiter. Ich habe mal spaßeshalber ein paar surrealistische Buntstiftzeichungen von mir hochgeladen und daraus neue in meinem Stil generieren lassen, das sieht nicht schlecht aus.