#conformalprediction — Public Fediverse posts

How do we make LLM output more trustworthy? A short survey note on three lines of recent work covering five papers: conformal-prediction coverage guarantees, behavioral calibration of the model's prose, and sample-disagreement detection. All three pay the same multi-sample inference tax; the choice is about what you want back.

https://benjaminhan.net/posts/20260505-llm-uncertainty-survey/?utm_source=mastodon&utm_medium=social

#Hallucination #LLMs #Calibration #ConformalPrediction #AI

Conformal Factuality casts LM correctness as uncertainty quantification. Decompose the answer into sub-claims, score each, drop the low-confidence ones until the retained set is ~1-α factual. The sub-claim decomposition is doing most of the work, and the conformal machinery rides on top. Atomic-claim splitters have known failure modes, and the guarantee inherits them.

https://benjaminhan.net/posts/20260505-conformal-factuality/?utm_source=mastodon&utm_medium=social

#ConformalPrediction #Calibration #Hallucination #LLMs #ICML #AI

Conformal Language Modeling (CLM) adapts conformal prediction to generative LMs: sample candidates, stop when a calibrated rule fires, return a set guaranteed to contain an acceptable answer. The more interesting half is the component-level filter — per-phrase coverage, not just set-level. That's the primitive for hallucination flagging: highlight the vetted phrases, leave the rest for review.

https://benjaminhan.net/posts/20260505-conformal-language-modeling/?utm_source=mastodon&utm_medium=social

#ConformalPrediction #LLMs #Hallucination #ICLR #AI

A primer on conformal prediction: the recipe for distribution-free coverage guarantees that doesn't require your model to be calibrated. Rank-based non-conformity scores plus a calibration quantile give you valid prediction sets. Easy inputs get one-class sets; hard ones get many alternatives. Set size is where the uncertainty shows up.

https://benjaminhan.net/posts/20260505-conformal-prediction-primer/?utm_source=mastodon&utm_medium=social

#ConformalPrediction #Calibration #UncertaintyQuantification #AI

🚨 New blog post 🚨

"**Optimal prediction sets for plant identification: an interactive guide**"

https://josephsalmon.eu/blog/long-tail/

Joint work with Tiffany Ding and Jean-Baptiste Fermanian.

#longtail
#PlantNet
#AppliedConformalPrediction
#ConformalPrediction

TIL of #conformalprediction, a way to assess the uncertainty of a prediction (from any algorithm, including from #machineleaning). It is used in research to make #autonomousdriving safer by predicting other agent's movements: https://www.youtube.com/watch?v=QvIJH4cZy3E

It does not require an expert model, but in turn it needs a statistically representative dataset.

🚀 #AWS Fortuna is skyrocketing! 🚀 Just a few days, and so many GitHub stars and forks! ⭐️

Fortuna supports #ConformalPrediction, #BayesianInference and other methods for #UncertaintyQuantification in #DeepLearning.

Try it out and let us know!
https://github.com/awslabs/fortuna

In collaboration with @cedapprox @andrewgwils and team.

#uncertainty #neuralnetworks #bayesian #conformal #calibration #jax #flax #python #opensource #library #machinelearning #ai

🚀 #AWS Fortuna is skyrocketing! 🚀 Just a few days, and so many GitHub stars and forks! ⭐️

Fortuna supports #ConformalPrediction, #BayesianInference and other methods for #UncertaintyQuantification in #DeepLearning.

Try it out and let us know!
https://github.com/awslabs/fortuna

In collaboration with @cedapprox, @andrewgwils and team.

#uncertainty #neuralnetworks #bayesian #conformal #calibration #jax #flax #python #opensource #library #machinelearning #ai