#aievaluation — Public Fediverse posts

ISD-Agent-Bench: A Comprehensive Benchmark for Evaluating LLM-based Instructional Design Agents
https://arxiv.org/abs/2602.10620
Code & data: https://github.com/codingchild2424/isd-agent-benchmark
"benchmark comprising 25,795 scenarios that combines 51 contextual variables across 5 categories with 33 ISD sub-steps derived from the ADDIE model."

w/same author: Pedagogy-R1: Pedagogical Large Reasoning Model and Well-balanced Educational Benchmark https://dl.acm.org/doi/10.1145/3746252.3761133
#AIEd #LearningDesign #AIevaluation #EdTech

ISD-Agent-Bench: A Comprehensive Benchmark for Evaluating LLM-based Instructional Design Agents
https://arxiv.org/abs/2602.10620
Code & data: https://github.com/codingchild2424/isd-agent-benchmark
"benchmark comprising 25,795 scenarios that combines 51 contextual variables across 5 categories with 33 ISD sub-steps derived from the ADDIE model."

w/same author: Pedagogy-R1: Pedagogical Large Reasoning Model and Well-balanced Educational Benchmark https://dl.acm.org/doi/10.1145/3746252.3761133
#AIEd #LearningDesign #AIevaluation #EdTech

ISD-Agent-Bench: A Comprehensive Benchmark for Evaluating LLM-based Instructional Design Agents
https://arxiv.org/abs/2602.10620
Code & data: https://github.com/codingchild2424/isd-agent-benchmark
"benchmark comprising 25,795 scenarios that combines 51 contextual variables across 5 categories with 33 ISD sub-steps derived from the ADDIE model."

w/same author: Pedagogy-R1: Pedagogical Large Reasoning Model and Well-balanced Educational Benchmark https://dl.acm.org/doi/10.1145/3746252.3761133
#AIEd #LearningDesign #AIevaluation #EdTech

ISD-Agent-Bench: A Comprehensive Benchmark for Evaluating LLM-based Instructional Design Agents
https://arxiv.org/abs/2602.10620
Code & data: https://github.com/codingchild2424/isd-agent-benchmark
"benchmark comprising 25,795 scenarios that combines 51 contextual variables across 5 categories with 33 ISD sub-steps derived from the ADDIE model."

w/same author: Pedagogy-R1: Pedagogical Large Reasoning Model and Well-balanced Educational Benchmark https://dl.acm.org/doi/10.1145/3746252.3761133
#AIEd #LearningDesign #AIevaluation #EdTech

ISD-Agent-Bench: A Comprehensive Benchmark for Evaluating LLM-based Instructional Design Agents
https://arxiv.org/abs/2602.10620
Code & data: https://github.com/codingchild2424/isd-agent-benchmark
"benchmark comprising 25,795 scenarios that combines 51 contextual variables across 5 categories with 33 ISD sub-steps derived from the ADDIE model."

w/same author: Pedagogy-R1: Pedagogical Large Reasoning Model and Well-balanced Educational Benchmark https://dl.acm.org/doi/10.1145/3746252.3761133
#AIEd #LearningDesign #AIevaluation #EdTech

https://winbuzzer.com/2026/04/06/google-study-ai-benchmarks-ignore-human-disagreement-xcxwbn/

Google Study: AI Benchmarks Use Too Few Raters to Be Reliable

#AI #Google #GoogleResearch #AIBenchmarks #AIResearch #MachineLearning #LMArena #ChatbotArena #BigTech #RochesterInstituteOfTechnology #AIEvaluation

Start your week off right with #enterpriseAI #changemanagement tips from IT leaders Juan Orlandini, Fabien CROS, Kulvir Gahunia and Dana Harrison. My in-depth look at how #gamification, #AIevaluation platforms, #platformengineering and other approaches helped companies such as Insight, Ducker Carlisle and TELUS adopt #AI effectively: https://www.techtarget.com/searchitoperations/news/366640354/IT-leaders-share-enterprise-AI-change-management-tips

#NLP #LLMs #MentalHealth #ClinicalNLP #DigitalHealth #ResponsibleAI #NLProc #AIevaluation #ModelEvaluation #TrustworthyAI #Safety #Equity #HumanCenteredAI

Một nhà phát triển vừa tạo công cụ đánh giá mã nguồn mở (SanityHarness) và kiểm tra 49 cặp mô hình/đại lý lập trình, bao gồm Kimi K2.5. Bảng xếp hạng SanityBoard chấm điểm hiệu năng, chi phí và so sánh các mô hình hỗ trợ BYOK. Phát hiện: Codebuff mắc nhưng hiệu suất kém, Droid và Minimax vượt trội. Mời cộng đồng tham gia thử nghiệm qua Discord. #AI #LậpTrình #ĐánhGiáAI #MãNguồnMở #Coding #AIEvaluation

https://www.reddit.com/r/LocalLLaMA/comments/1qp4ftj/i_made_a_coding_eval_and_ran_it_against_4

TrustifAI – Khung đánh giá độ tin cậy cho hệ thống AI/RAG với điểm số đa chiều: Độ bao phủ bằng chứng, Độ ổn định luận lý, Độ lệch ngữ nghĩa, Đa dạng nguồn, Độ tự tin sinh nội dung. Tạo đồ thị lý lẽ & trực quan hóa Mermaid để truy xuất nguyên nhân. Giải pháp cho môi trường doanh nghiệp, quản trị & tuân thủ. #TrustifAI #RAG #AIEvaluation #AIinVietnam #ĐánhGiáAI #HệThốngThôngMinh

https://www.reddit.com/gallery/1qmhvuz

Sự phát triển nhanh chóng của các mô hình AI hiện đại đòi hỏi bộ tiêu chuẩn đánh giá sâu rộng năng lực phức tạp, nhằm thúc đẩy hoàn thiện các mô hình ngôn ngữ lớn (LLM) tiên tiến. Các chuyên gia nhấn mạnh, AI càng thông minh, việc đánh giá càng phải toàn diện hơn để đảm bảo an toàn và hiệu quả.

#AI #TríTuệNhânTạo #AIModels #MôHìnhAI #AIEvaluation #ĐánhGiáAI #CôngNghe #Tech

https://vietnamnet.vn/cang-thong-minh-mo-hinh-ai-cang-can-bo-tieu-chuan-danh-gia-nang-luc-phuc-tap-2448553.html

Why accessibility might be AI’s biggest breakthrough - While tech companies market AI as a productivity tool for ev... - https://arstechnica.com/information-technology/2025/09/study-finds-neurodiverse-workers-more-satisfied-with-ai-assistants/ #departmentforbusinessandtrade #workplaceaccommodation #aiaccessibility #machinelearning #neurodiversity #accessibility #aiassistants #aievaluation #ukgovernment #m365copilot #disability #aiandwork #microsoft #dyslexia #aistudy #biz⁢ #adhd #ai

The educator panic over AI is real, and rational.
I've been there myself. The difference is I moved past denial to a more pragmatic question: since AI regulation seems unlikely (with both camps refusing to engage), how do we actually work with these systems?

The "AI will kill critical thinking" crowd has a point, but they're missing context.
Critical reasoning wasn't exactly thriving before AI arrived: just look around. The real question isn't whether AI threatens thinking skills, but whether we can leverage it the same way we leverage other cognitive tools.

We don't hunt our own food or walk everywhere anymore.
We use supermarkets and cars. Most of us Google instead of visiting libraries. Each tool trade-off changed how we think and what skills matter. AI is the next step in this progression, if we're smart about it.

The key is learning to think with AI rather than being replaced by it.
That means understanding both its capabilities and our irreplaceable human advantages.

1/3

#AI #Education #FutureOfEducation #AIinEducation #LLM #ChatGPT #Claude #EdAI #CriticalThinking #CognitiveScience #Metacognition #HigherOrderThinking #Reasoning #Vygotsky #Hutchins #Sweller #LearningScience #EducationalPsychology #SocialLearning #TechforGood #EticalAI #AILiteracy #PromptEngineering #AISkills #DigitalLiteracy #FutureSkills #LRM #AIResearch #AILimitations #SystemsThinking #AIEvaluation #MentalModels #LifelongLearning #AIEthics #HumanCenteredAI #DigitalTransformation #AIRegulation #ResponsibleAI #Philosophy

AI isn't going anywhere. Time to get strategic:
Instead of mourning lost critical thinking skills, let's build on them through cognitive delegation—using AI as a thinking partner, not a replacement.

This isn't some Silicon Valley fantasy:
Three decades of cognitive research already mapped out how this works:

Cognitive Load Theory:
Our brains can only juggle so much at once. Let AI handle the grunt work while you focus on making meaningful connections.

Distributed Cognition:
Naval crews don't navigate with individual genius—they spread thinking across people, instruments, and procedures. AI becomes another crew member in your cognitive system.

Zone of Proximal Development
We learn best with expert guidance bridging what we can't quite do alone. AI can serve as that "more knowledgeable other" (though it's still early days).
The table below shows what this looks like in practice:

2/3

#AI #Education #FutureOfEducation #AIinEducation #LLM #ChatGPT #Claude #EdAI #CriticalThinking #CognitiveScience #Metacognition #HigherOrderThinking #Reasoning #Vygotsky #Hutchins #Sweller #LearningScience #EducationalPsychology #SocialLearning #TechforGood #EticalAI #AILiteracy #PromptEngineering #AISkills #DigitalLiteracy #FutureSkills #LRM #AIResearch #AILimitations #SystemsThinking #AIEvaluation #MentalModels #LifelongLearning #AIEthics #HumanCenteredAI #DigitalTransformation #AIRegulation #ResponsibleAI #Philosophy

Critical reasoning vs Cognitive Delegation

Old School Focus:

Building internal cognitive capabilities and managing cognitive load independently.

Cognitive Delegation Focus:

Orchestrating distributed cognitive systems while maintaining quality control over AI-augmented processes.

We can still go for a jog or go hunt our own deer, but for reaching the stars we, the Apes do what Apes do best: Use tools to build on our cognitive abilities. AI is a tool.

3/3

#AI #Education #FutureOfEducation #AIinEducation #LLM #ChatGPT #Claude #EdAI #CriticalThinking #CognitiveScience #Metacognition #HigherOrderThinking #Reasoning #Vygotsky #Hutchins #Sweller #LearningScience #EducationalPsychology #SocialLearning #TechforGood #EticalAI #AILiteracy #PromptEngineering #AISkills #DigitalLiteracy #FutureSkills #LRM #AIResearch #AILimitations #SystemsThinking #AIEvaluation #MentalModels #LifelongLearning #AIEthics #HumanCenteredAI #DigitalTransformation #AIRegulation #ResponsibleAI #Philosophy

Master Python Ragas AI Evaluation! Learn to effectively assess your LLMs and RAG systems for top-tier performance. Full tutorial inside. #Python #Ragas #AIEvaluation #LLM #RAG #TechTutorial #DataScience

https://teguhteja.id/python-ragas-ai-evaluation-master-llm-assessment-guide/

Experts Challenge Validity and Ethics of Crowdsourced AI Benchmarks Like LMArena (Chatbot Arena)

#AI #AIBenchmarks #AIModels #LMArena #ChatbotArena #AIethics #LLMs #AIEvaluation #Crowdsourcing #GenAI

https://winbuzzer.com/2025/04/22/experts-challenge-validity-and-ethics-of-crowdsourced-ai-benchmarks-like-lmarena-chatbot-arena-xcxwbn/

AI Benchmarking Platform Chatbot Arena Forms New Company, Launches LMArena

#AI #GenAI #LLMs #AIChatbots #LMArena #ChatbotArena #AIBenchmarks #AIModels #AIevaluation

https://winbuzzer.com/2025/04/18/ai-benchmarking-platform-chatbot-arena-forms-new-company-launches-lmarena-xcxwbn/

ICYMI: Google updates quality rater guidelines with AI content evaluation criteria: Google's latest guidelines provide clearer direction on evaluating AI-generated content and spam tactics. https://ppc.land/google-updates-quality-rater-guidelines-with-ai-content-evaluation-criteria/ #GoogleUpdates #QualityRater #AIEvaluation #ContentGuidelines #DigitalMarketing

#AI #interpretability vs #explainability 🧵

"The explanations themselves can be difficult to convey to nonexperts, such as end users and line-of-business teams" https://www.techtarget.com/searchenterpriseai/feature/Interpretability-vs-explainability-in-AI-and-machine-learning

#AIEthics #compliance #taxonomy #ethicalAI #AIEvaluation #linearRegression #trust #neuralNetworks #ML #governance #AIgovernance #safety #bias

"Feature importance helps in understanding which features contribute most to the prediction"

A few lines with #sklearn: https://mljourney.com/sklearn-linear-regression-feature-importance/

#interpretability #explainability #AIethics #compliance #taxonomy #ethicalAI #AIevaluation #linearRegression #featureEngineering

"Feature importance helps in understanding which features contribute most to the prediction"

A few lines with #sklearn: https://mljourney.com/sklearn-linear-regression-feature-importance/

#interpretability #explainability #AIethics #compliance #taxonomy #ethicalAI #AIevaluation #linearRegression #featureEngineering

"Feature importance helps in understanding which features contribute most to the prediction"

A few lines with #sklearn: https://mljourney.com/sklearn-linear-regression-feature-importance/

#interpretability #explainability #AIethics #compliance #taxonomy #ethicalAI #AIevaluation #linearRegression #featureEngineering

"Feature importance helps in understanding which features contribute most to the prediction"

A few lines with #sklearn: https://mljourney.com/sklearn-linear-regression-feature-importance/

#interpretability #explainability #AIethics #compliance #taxonomy #ethicalAI #AIevaluation #linearRegression #featureEngineering

"Feature importance helps in understanding which features contribute most to the prediction"

A few lines with #sklearn: https://mljourney.com/sklearn-linear-regression-feature-importance/

#interpretability #explainability #AIethics #compliance #taxonomy #ethicalAI #AIevaluation #linearRegression #featureEngineering

"The #gamma GLM is a relatively assumption-light means of #modeling non-negative data, given gamma's flexibility.
[…]
"Explaining what is used and what is not used, despite merits and demerits […]: Loosely, the larger the internal literature in any field on modelling techniques, the less inclined people in that field seem to be to try something different."

Nick Cox, 2013: https://stats.stackexchange.com/questions/67547/when-to-use-gamma-glms

#normality #normalDistribution #Γ #modelling #dataDev #AIDev #ML #AIEvaluation #logNormal

"The #gamma GLM is a relatively assumption-light means of #modeling non-negative data, given gamma's flexibility.
[…]
"Explaining what is used and what is not used, despite merits and demerits […]: Loosely, the larger the internal literature in any field on modelling techniques, the less inclined people in that field seem to be to try something different."

Nick Cox, 2013: https://stats.stackexchange.com/questions/67547/when-to-use-gamma-glms

#normality #normalDistribution #Γ #modelling #dataDev #AIDev #ML #AIEvaluation #logNormal

"The #gamma GLM is a relatively assumption-light means of #modeling non-negative data, given gamma's flexibility.
[…]
"Explaining what is used and what is not used, despite merits and demerits […]: Loosely, the larger the internal literature in any field on modelling techniques, the less inclined people in that field seem to be to try something different."

Nick Cox, 2013: https://stats.stackexchange.com/questions/67547/when-to-use-gamma-glms

#normality #normalDistribution #Γ #modelling #dataDev #AIDev #ML #AIEvaluation #logNormal

"The #gamma GLM is a relatively assumption-light means of #modeling non-negative data, given gamma's flexibility.
[…]
"Explaining what is used and what is not used, despite merits and demerits […]: Loosely, the larger the internal literature in any field on modelling techniques, the less inclined people in that field seem to be to try something different."

Nick Cox, 2013: https://stats.stackexchange.com/questions/67547/when-to-use-gamma-glms

#normality #normalDistribution #Γ #modelling #dataDev #AIDev #ML #AIEvaluation #logNormal

"The #gamma GLM is a relatively assumption-light means of #modeling non-negative data, given gamma's flexibility.
[…]
"Explaining what is used and what is not used, despite merits and demerits […]: Loosely, the larger the internal literature in any field on modelling techniques, the less inclined people in that field seem to be to try something different."

Nick Cox, 2013: https://stats.stackexchange.com/questions/67547/when-to-use-gamma-glms

#normality #normalDistribution #Γ #modelling #dataDev #AIDev #ML #AIEvaluation #logNormal

@datadon

"The following sections discuss several state-of-the-art interpretable and explainable #ML methods. The selection of works does not comprise an exhaustive survey of the literature. Instead, it is meant to illustrate the commonest properties and inductive biases behind interpretable models and [black-box] explanation methods using concrete instances."
https://wires.onlinelibrary.wiley.com/doi/full/10.1002/widm.1493#widm1493-sec-0010-title 🧵

#interpretability #explainability #aiethics #compliance #taxonomy #ethicalai #aievaluation #linearRegression

Model "#interpretability and [black-box] #explainability, although not necessary in many straightforward applications, become instrumental when the problem definition is incomplete and in the presence of additional desiderata, such as trust, causality, or fairness."

https://wires.onlinelibrary.wiley.com/doi/full/10.1002/widm.1493

#aiethics #compliance #taxonomy #ethicalai #aievaluation