Search
1000 results for “Benja”
-
How do we make LLM output more trustworthy? A short survey note on three lines of recent work covering five papers: conformal-prediction coverage guarantees, behavioral calibration of the model's prose, and sample-disagreement detection. All three pay the same multi-sample inference tax; the choice is about what you want back.
https://benjaminhan.net/posts/20260505-llm-uncertainty-survey/?utm_source=mastodon&utm_medium=social
-
How do we make LLM output more trustworthy? A short survey note on three lines of recent work covering five papers: conformal-prediction coverage guarantees, behavioral calibration of the model's prose, and sample-disagreement detection. All three pay the same multi-sample inference tax; the choice is about what you want back.
https://benjaminhan.net/posts/20260505-llm-uncertainty-survey/?utm_source=mastodon&utm_medium=social
-
How do we make LLM output more trustworthy? A short survey note on three lines of recent work covering five papers: conformal-prediction coverage guarantees, behavioral calibration of the model's prose, and sample-disagreement detection. All three pay the same multi-sample inference tax; the choice is about what you want back.
https://benjaminhan.net/posts/20260505-llm-uncertainty-survey/?utm_source=mastodon&utm_medium=social
-
Costa Rica / Israel: ¿Benjamín Netanyahu… asistiendo al próximo acto de traspaso de poderes en Costa Rica?
Costa Rica / Israel: ¿Benjamín Netanyahu… asistiendo al próximo acto de traspaso de poderes en Costa Rica?
El pasado 27 de abril, un titular de un artículo de La Nación se lee como sigue: “¿Vendrá el presidente o el primer ministro de Israel a la toma de posesión de L [...]#BenjaminNetanyahu #Gaza #Israel #LauraFernández #Opinión #RodrigoChaves #TraspasoDePoderes
-
Semantic Entropy (Nature 2024) detects LLM confabulations by clustering sampled answers by meaning and computing entropy over the cluster distribution. "Paris" and "It's Paris" cluster together, so paraphrase noise doesn't inflate the signal. Cost: it only catches hallucinations that vary across samples. If the model is consistently wrong, all samples cluster and the detector says "confident".
https://benjaminhan.net/posts/20260505-semantic-entropy/?utm_source=mastodon&utm_medium=social
-
Semantic Entropy (Nature 2024) detects LLM confabulations by clustering sampled answers by meaning and computing entropy over the cluster distribution. "Paris" and "It's Paris" cluster together, so paraphrase noise doesn't inflate the signal. Cost: it only catches hallucinations that vary across samples. If the model is consistently wrong, all samples cluster and the detector says "confident".
https://benjaminhan.net/posts/20260505-semantic-entropy/?utm_source=mastodon&utm_medium=social
-
Semantic Entropy (Nature 2024) detects LLM confabulations by clustering sampled answers by meaning and computing entropy over the cluster distribution. "Paris" and "It's Paris" cluster together, so paraphrase noise doesn't inflate the signal. Cost: it only catches hallucinations that vary across samples. If the model is consistently wrong, all samples cluster and the detector says "confident".
https://benjaminhan.net/posts/20260505-semantic-entropy/?utm_source=mastodon&utm_medium=social
-
Semantic Entropy (Nature 2024) detects LLM confabulations by clustering sampled answers by meaning and computing entropy over the cluster distribution. "Paris" and "It's Paris" cluster together, so paraphrase noise doesn't inflate the signal. Cost: it only catches hallucinations that vary across samples. If the model is consistently wrong, all samples cluster and the detector says "confident".
https://benjaminhan.net/posts/20260505-semantic-entropy/?utm_source=mastodon&utm_medium=social
-
Semantic Entropy (Nature 2024) detects LLM confabulations by clustering sampled answers by meaning and computing entropy over the cluster distribution. "Paris" and "It's Paris" cluster together, so paraphrase noise doesn't inflate the signal. Cost: it only catches hallucinations that vary across samples. If the model is consistently wrong, all samples cluster and the detector says "confident".
https://benjaminhan.net/posts/20260505-semantic-entropy/?utm_source=mastodon&utm_medium=social
-
Linguistic Calibration trains Llama 2 to emit confidence phrases that let a downstream reader make calibrated forecasts on related questions. The key move is defining calibration through reader utility instead of self-reported probability. Hedged text that doesn't help the reader makes no forecasting progress, so generic hedging can't game the objective.
https://benjaminhan.net/posts/20260505-linguistic-calibration/?utm_source=mastodon&utm_medium=social
-
Linguistic Calibration trains Llama 2 to emit confidence phrases that let a downstream reader make calibrated forecasts on related questions. The key move is defining calibration through reader utility instead of self-reported probability. Hedged text that doesn't help the reader makes no forecasting progress, so generic hedging can't game the objective.
https://benjaminhan.net/posts/20260505-linguistic-calibration/?utm_source=mastodon&utm_medium=social
-
Linguistic Calibration trains Llama 2 to emit confidence phrases that let a downstream reader make calibrated forecasts on related questions. The key move is defining calibration through reader utility instead of self-reported probability. Hedged text that doesn't help the reader makes no forecasting progress, so generic hedging can't game the objective.
https://benjaminhan.net/posts/20260505-linguistic-calibration/?utm_source=mastodon&utm_medium=social
-
Linguistic Calibration trains Llama 2 to emit confidence phrases that let a downstream reader make calibrated forecasts on related questions. The key move is defining calibration through reader utility instead of self-reported probability. Hedged text that doesn't help the reader makes no forecasting progress, so generic hedging can't game the objective.
https://benjaminhan.net/posts/20260505-linguistic-calibration/?utm_source=mastodon&utm_medium=social
-
Linguistic Calibration trains Llama 2 to emit confidence phrases that let a downstream reader make calibrated forecasts on related questions. The key move is defining calibration through reader utility instead of self-reported probability. Hedged text that doesn't help the reader makes no forecasting progress, so generic hedging can't game the objective.
https://benjaminhan.net/posts/20260505-linguistic-calibration/?utm_source=mastodon&utm_medium=social
-
Conformal Factuality casts LM correctness as uncertainty quantification. Decompose the answer into sub-claims, score each, drop the low-confidence ones until the retained set is ~1-α factual. The sub-claim decomposition is doing most of the work, and the conformal machinery rides on top. Atomic-claim splitters have known failure modes, and the guarantee inherits them.
https://benjaminhan.net/posts/20260505-conformal-factuality/?utm_source=mastodon&utm_medium=social
#ConformalPrediction #Calibration #Hallucination #LLMs #ICML #AI
-
Conformal Factuality casts LM correctness as uncertainty quantification. Decompose the answer into sub-claims, score each, drop the low-confidence ones until the retained set is ~1-α factual. The sub-claim decomposition is doing most of the work, and the conformal machinery rides on top. Atomic-claim splitters have known failure modes, and the guarantee inherits them.
https://benjaminhan.net/posts/20260505-conformal-factuality/?utm_source=mastodon&utm_medium=social
#ConformalPrediction #Calibration #Hallucination #LLMs #ICML #AI
-
Conformal Factuality casts LM correctness as uncertainty quantification. Decompose the answer into sub-claims, score each, drop the low-confidence ones until the retained set is ~1-α factual. The sub-claim decomposition is doing most of the work, and the conformal machinery rides on top. Atomic-claim splitters have known failure modes, and the guarantee inherits them.
https://benjaminhan.net/posts/20260505-conformal-factuality/?utm_source=mastodon&utm_medium=social
#ConformalPrediction #Calibration #Hallucination #LLMs #ICML #AI
-
Conformal Factuality casts LM correctness as uncertainty quantification. Decompose the answer into sub-claims, score each, drop the low-confidence ones until the retained set is ~1-α factual. The sub-claim decomposition is doing most of the work, and the conformal machinery rides on top. Atomic-claim splitters have known failure modes, and the guarantee inherits them.
https://benjaminhan.net/posts/20260505-conformal-factuality/?utm_source=mastodon&utm_medium=social
#ConformalPrediction #Calibration #Hallucination #LLMs #ICML #AI
-
Conformal Factuality casts LM correctness as uncertainty quantification. Decompose the answer into sub-claims, score each, drop the low-confidence ones until the retained set is ~1-α factual. The sub-claim decomposition is doing most of the work, and the conformal machinery rides on top. Atomic-claim splitters have known failure modes, and the guarantee inherits them.
https://benjaminhan.net/posts/20260505-conformal-factuality/?utm_source=mastodon&utm_medium=social
#ConformalPrediction #Calibration #Hallucination #LLMs #ICML #AI
-
Conformal Language Modeling (CLM) adapts conformal prediction to generative LMs: sample candidates, stop when a calibrated rule fires, return a set guaranteed to contain an acceptable answer. The more interesting half is the component-level filter — per-phrase coverage, not just set-level. That's the primitive for hallucination flagging: highlight the vetted phrases, leave the rest for review.
-
Conformal Language Modeling (CLM) adapts conformal prediction to generative LMs: sample candidates, stop when a calibrated rule fires, return a set guaranteed to contain an acceptable answer. The more interesting half is the component-level filter — per-phrase coverage, not just set-level. That's the primitive for hallucination flagging: highlight the vetted phrases, leave the rest for review.
-
Conformal Language Modeling (CLM) adapts conformal prediction to generative LMs: sample candidates, stop when a calibrated rule fires, return a set guaranteed to contain an acceptable answer. The more interesting half is the component-level filter — per-phrase coverage, not just set-level. That's the primitive for hallucination flagging: highlight the vetted phrases, leave the rest for review.
-
Conformal Language Modeling (CLM) adapts conformal prediction to generative LMs: sample candidates, stop when a calibrated rule fires, return a set guaranteed to contain an acceptable answer. The more interesting half is the component-level filter — per-phrase coverage, not just set-level. That's the primitive for hallucination flagging: highlight the vetted phrases, leave the rest for review.
-
Conformal Language Modeling (CLM) adapts conformal prediction to generative LMs: sample candidates, stop when a calibrated rule fires, return a set guaranteed to contain an acceptable answer. The more interesting half is the component-level filter — per-phrase coverage, not just set-level. That's the primitive for hallucination flagging: highlight the vetted phrases, leave the rest for review.
-
A primer on conformal prediction: the recipe for distribution-free coverage guarantees that doesn't require your model to be calibrated. Rank-based non-conformity scores plus a calibration quantile give you valid prediction sets. Easy inputs get one-class sets; hard ones get many alternatives. Set size is where the uncertainty shows up.
#ConformalPrediction #Calibration #UncertaintyQuantification #AI
-
A primer on conformal prediction: the recipe for distribution-free coverage guarantees that doesn't require your model to be calibrated. Rank-based non-conformity scores plus a calibration quantile give you valid prediction sets. Easy inputs get one-class sets; hard ones get many alternatives. Set size is where the uncertainty shows up.
#ConformalPrediction #Calibration #UncertaintyQuantification #AI
-
A primer on conformal prediction: the recipe for distribution-free coverage guarantees that doesn't require your model to be calibrated. Rank-based non-conformity scores plus a calibration quantile give you valid prediction sets. Easy inputs get one-class sets; hard ones get many alternatives. Set size is where the uncertainty shows up.
#ConformalPrediction #Calibration #UncertaintyQuantification #AI
-
A primer on conformal prediction: the recipe for distribution-free coverage guarantees that doesn't require your model to be calibrated. Rank-based non-conformity scores plus a calibration quantile give you valid prediction sets. Easy inputs get one-class sets; hard ones get many alternatives. Set size is where the uncertainty shows up.
#ConformalPrediction #Calibration #UncertaintyQuantification #AI
-
A primer on conformal prediction: the recipe for distribution-free coverage guarantees that doesn't require your model to be calibrated. Rank-based non-conformity scores plus a calibration quantile give you valid prediction sets. Easy inputs get one-class sets; hard ones get many alternatives. Set size is where the uncertainty shows up.
#ConformalPrediction #Calibration #UncertaintyQuantification #AI
-
Benjamin Libet: If your brain initiates action before your consciousness of deciding, then what exactly is the 'you' that thinks it is choosing? https://www.youtube.com/watch?v=61nAQFREfYM #Freewill #neuroscience