#mechanisticinterpretability — Public Fediverse posts

Morning, today we're looking at #MechanisticInterpretability , a subfield of AI Safety that attempts to understand the inner workings of artificial intelligence by analysing concrete structures, algorithms and circuits.

Why do we even need to do this? Because the hope of understanding neurons as being features died on polysemanticity -- models represent more features than dimensions by assigning them to an overcomplete set of non-orthogonal directions (i.e. you can't hope that concepts can be broken down into linear combinations of features).

This isn't a new field, people have been projecting intermediate GPT layers through the final layer activation and looking at the top-k most likely next tokens since GPT-2.

And that's how we arrive at the Natural Language Autoencoder -- an autoencoder over residual-stream activations where the bottleneck is natural language text instead of a sparse vector.

Nothing in the training objective requires the verbalisation bottleneck to be readable, faithful, or even semantically related to the intermediate layer being investigated, so while the training algorithm is typical of autoencoders, two things are distinct here:

1) both parts of the autoencoder (here the activation-verbaliser, AV, and the activation-reconstructor, AR) undergo special SFT to ensure the verbaliser and reconstructor can generate and read natural language from intermediate layer activations
2) a KL-divergence penalty is baked into the objective to make sure that, during joint training, the NLA doesn't diverge from the SFT version

A side-product of this training is that we can take the current verbalisation and "desired" verbalisation and compute steering vectors from their difference.

Surprisingly, Anthropic didn't find any evidence that the NLA was engaging in steganography to smuggle information to avoid human detection, but it did find that NLAs tend to confabulate a lot.

#AIResearch #Transformers #TransformerCircuits #Anthropic #NLA

Calling LLMs stochastic parrots is... frankly wrong. It's true that pre-training LLMs is literally matching the empirical distribution of human text, so this would be the only place where "stochastic parrot" is a coherent description of the objective. The capabilities that emerge when training for this objective are anything but parrotry.

We've empirically proved that when learning high dimensional datasets, the interpolation that "stochastic parrotry" implies never happens, and extrapolation actually is the result (arXiv:2110.09485 [cs.LG]). Other research also shows out of distribution generalisation of transformers (Othello-GPT arXiv:2210.13382 [cs.LG] shows that a transformer trained purely on move sequences develops a linear, causally-manipulable representation of board state).

#AI #AIResearch #Transformers #LLM #MechanisticInterpretability

Learn more about Phu and his work: https://phusroyal.github.io/

Welcome to the team, Phu! 👋

#UKPLab #TUDarmstadt #MBZUAI #NLP #NLProc #MechanisticInterpretability #LLMs #AIInterpretability

Questions? Discussion? Reach out to us:

Andreas Waldis (UKP Lab/Technische Universität Darmstadt and HSLU Hochschule Luzern), Vagrant Gautam (Universität des Saarlandes), Anne Lauscher (Universität Hamburg), Dietrich Klakow (Universität des Saarlandes), and Iryna Gurevych (UKP Lab/Technische Universität Darmstadt)

#NLProc #Interpretability #LLMs #ExplainableAI #MechanisticInterpretability #AlignedProbing #ModelInternals

Gemma Scope Empowers AI Safety Community with Model Transparency

https://techlife.blog/posts/gemma-scope/

#AISafety
#DeepMind
#Gemma
#MechanisticInterpretability
#AIInterpretability

[Перевод] Как сделать нейросети понятнее: эксперимент OpenAI с разряженными моделями

Команда AI for Devs подготовила перевод исследования OpenAI о том, как обучение разреженных моделей может сделать ИИ более прозрачным. Авторы показывают: если заставить модель использовать меньше связей, внутри неё появляются понятные цепочки вычислений, которые можно изучать и проверять. Это может стать шагом к созданию мощных, но интерпретируемых систем.

https://habr.com/ru/articles/966448/

#интерпретируемость #разреженныемодели #mechanisticinterpretability #sparsetransformer #цепочкивычислений #circuits #OpenAI #безопасностьИИ #attention #архитектурамоделей

"But every once in a while, Claude breaks bad. It lies. It deceives. It develops weird obsessions. It makes threats and then carries them out. And the frustrating part—true of all LLMs—is that no one knows exactly why." @stevenlevy for Wired

https://www.wired.com/story/ai-black-box-interpretability-problem/

#AI #LLMs #MechanisticInterpretability

Can someone find me a job doable from Amsterdam in mechanistic interpretability? #AI #MI #MechanisticInterpretability

@AAKL @NGIZero @Reuters @EC_NGI

Trying to regulate AI can be like regulating math in that suddenly certain calculations are illegal.

Trying to regulate AI can be like regulating the printing press in that suddenly only people with enough lawyers are able to make a printing press.

Trying to regulate AI can be like trying to regulate free speech. From now on only certain forms of speech are allowed (a bit like the book 1984).

Microsoft, Facebook, Google, Nvidia and others are investing billions (trillions?). Big Tech and Governments want to continue with data mining, government surveillance and surveillance capitalism. This will go as long as long as we let them. The incentives are there. To change that we need social awareness, a consciousness shift or a libre ethical digital revolution.

If you truly want to stop AI then think about how to organize:

• anti tech. See “anti-tech revolution why and how”
• anti AI / anti AGI. see https://betterwithout.ai

Otherwise focusing on research, libre AI, libre riscv, libre silicon and research (ethics,safety,mechanistic interpretability) is currently our best hope.

#betterwithoutai #antitechrevolution #libreai #libreriscv #libresilicon #aiethics #ethicalai #aisafety #mechanisticinterpretability #riscv #ai #datamining #governmentsurveillance

@AAKL @Reuters

Instead of making laws focus on funding:

• libre AI
• libre risc v
• libre silicon
• research on AI ethics
• research on AI safety
• research on mechanistic interpretability
• NLnet
• projects like openchatkit and open asisstant

Laws and regulations (including license restrictions) usually benefit corporations and governments who have lawyers, lobyists, perverse incentives and a drive for data mining and surveillance. Even if you do your best to make a good law it could make it harder for libre efforts to comply. Put your trust in research, decentralization, libre software, libre hardware and research.

Billions of dollars are being invested by companies like nvidia,google and microsoft.

#libreai #nlnet #libreai #libreriscv #riscv #libresilicon #aiethics #ethicalai #aisafety #mechanisticinterpretability #freelibresoftware #libresoftware

@NGIZero @EC_NGI

@gregorni

Related efforts that I know of.

The ability to run bloom on your own hardware: https://github.com/NouamaneTazi/bloomz.cpp https://github.com/NouamaneTazi/bloomz.cpp/issues/4

Vipergpt. Not sure whether it will be libre. (Hopefully) https://viper.cs.columbia.edu/

King Algorithm Manifesto https://github.com/keskival/king-algorithm-manifesto/blob/main/README.md

Better Without AI which advocates for mechanistic interpretability https://betterwithout.ai

There is also a recommended reading list for stochastic parrots but I am not linking it here since I do not want to promote google docs.

Also does anyone know of any efforts to provide libre riscv AI accelerator that can be run on FPGA?

#bloom #vipergpt #kingalgorithmmanifesto #stochasticparrots #betterwithoutai #mechanisticinterpretability #riscv

@tero @simon @estranho @emilymbender

Thanks for the interesting discussion.

IMHO fully open language models (including the weights) under a libre license (without restrictions) in combination with more research on mechanistic interpretability might eventually lead to useful libre models running on our own hardware.

https://duckduckgo.com/?q=mechanistic+interpretability+site%3Ahttps%3A%2F%2Fbetterwithout.ai&ia=web

Question to all: is there anything special about chatgpt4 compared to chatgpt (3.5) ?

#chatgpt4 #gpt4 #openai #mechanisticinterpretability