I ran the same job past three different AI providers and had them vote on the answer. For a hobby project about a Sunday night D&D game.
Extravagant, I know, until you look at the job. Scout listens to a session and turns it into campaign facts, and every proper noun in that transcript was invented that week by someone sat at the table. No world knowledge to fall back on.
And extraction is a write, not a read. A wrong fact sits in the store being wrong, looking exactly as confident as the right ones.
The three models didn't fail more or less than each other, they failed in different directions. That's what a panel is for. Three copies of one model just give you the same answer three times with a rosette on it!
The voting mechanism is about forty lines. What cost me a weekend was working out when two facts are the same fact. Two models, the same thirty seconds of play, the same three invented names... and each had bundled different extra facts into the statement.
I thought the problem was picking the right model. It was working out what a fact is.
#LLM #DnD #TTRPG #SoftwareEngineering
https://phpboyscout.uk/three-models-vote-on-what-happened/