OrcaCyber-Nano came out with results on 12 different benchmarks, just to say what it does.
I thought a benchmark was meant to be a standard. Now there are dozens, some easy, some brutal.
And once a benchmark's prompts are public, a model can be trained on exactly those prompts. So maybe closed benchmarks are next. But then: can I be trusted with your weights, or you with my prompts?
I'm building my own, so I know how hard this is. www.senbon.dev #AI #LLM
A language model can refuse you, flatter you, or insult you. Three behaviours, and every one of them has a human twin.
Refusal is the famous one. You can find it inside a model and remove it.
I've spent weeks measuring the other two: sycophancy, how willing a model is to agree with you, and toxicity, whether it's rude to you or worse.
Both look a lot like human habits, and I don't have an opinion yet on what that means. The waitlist is at www.senbon.dev