#llmbenchmarking — Public Fediverse posts

RT @witcheer: GPT-OSS-20B im Detail: 8 Coding-Aufgaben, 8 Durchläufe, 1,8 GB VRAM. Einrichtung: RTX 4060 Ti 8GB, WSL2, llama-server mit ncmoe=30, Pi Coding Agent. Das Modell nutzt 1,8 GB VRAM und 10 GB Host-RAM. Ergebnis: 8/8 bestanden. Jede Aufgabe produzierte funktionierenden, getesteten Code. Zusammen mit dem ursprünglichen Benchmark (portscout + logpulse) sind das 10/10 abgeschlossene agentic Aufgaben auf Consumer-Hardware. (alle Prompts finden Sie auf Hugging Face). Was ich herausfand: Kontexteffizienz: 6-48 % des 32K-Fensters pro Aufgabe genutzt. Keine Aufgabe erschöpfte den Kontext. Der schwierigste Prompt (Multi-Module mit topologischer Sortierung) nutzte 47,6 %. Selbstkorrektur funktioniert: Das Modell fand und behob seine eigenen Fehler 7 Mal über 8 Aufgaben hinweg. Die Richtung der topologischen Sortierung war umgekehrt, behoben. printf-Formatzeichenfolge fehlte %s, behoben. sed-Zitierung war falsch, behoben. Keine halluzinierten APIs: Prompt 5 war eine Falle: „Verwenden Sie nur die Standardbibliothek.“ Das Modell nutzte echte Module (http.client, json, time, urllib.parse). Keine erfundenen Convenience-Wrapper. Gefundene Schwächen: Edit-Tool hat Schwierigkeiten: Exakte String-Matching für das Edit-Tool schlägt wiederholt fehl. Das Modell benötigt 3-4 Versuche, bevor es auf vollständige Datei-Neuschreibungen zurückgreift. Dies ist teilweise ein Problem auf Agent-Seite (Pi's Edit-Tool erfordert exaktes Matching), nicht rein ein Modellproblem. Verschwendung bei Verzeichnis-Scans: Führt „ls -R ~“ vom Home-Verzeichnis aus und spuckt 11K+ Zeilen in den Kontext aus. Passierte bei 2 v…

mehr auf Arint.info

#AIResearch #CodingAgent #GPTOSS #LLMbenchmarking #MoeArchitecture #Quantization #arint_info

https://x.com/witcheer/status/2056830610044490020#m

Deep dive analysis of Grok 4.2 and Sonnet 4.6, two new AI releases from xAI and Anthropic, and how their agent systems compare. https://hackernoon.com/grok-42-vs-sonnet-46-early-impressions-from-hands-on-testing #llmbenchmarking

Deep dive analysis of Grok 4.2 and Sonnet 4.6, two new AI releases from xAI and Anthropic, and how their agent systems compare. https://hackernoon.com/grok-42-vs-sonnet-46-early-impressions-from-hands-on-testing #llmbenchmarking

Deep dive analysis of Grok 4.2 and Sonnet 4.6, two new AI releases from xAI and Anthropic, and how their agent systems compare. https://hackernoon.com/grok-42-vs-sonnet-46-early-impressions-from-hands-on-testing #llmbenchmarking

Deep dive analysis of Grok 4.2 and Sonnet 4.6, two new AI releases from xAI and Anthropic, and how their agent systems compare. https://hackernoon.com/grok-42-vs-sonnet-46-early-impressions-from-hands-on-testing #llmbenchmarking

Deep dive analysis of Grok 4.2 and Sonnet 4.6, two new AI releases from xAI and Anthropic, and how their agent systems compare. https://hackernoon.com/grok-42-vs-sonnet-46-early-impressions-from-hands-on-testing #llmbenchmarking

Qwen3‑Coder‑Next slashes through the competition, delivering 10× the throughput of Claude‑Opus‑4.5 on SecCodeBench’s repository‑level tasks. The open‑source model not only speeds up AI code generation but also boosts vulnerability detection. Dive into the benchmark details and see why it’s a game‑changer for secure coding. #Qwen3CoderNext #SecCodeBench #LLMBenchmarking #OpenSourceAI

🔗 https://aidailypost.com/news/qwen3-coder-next-10-throughput-beats-claudeopus45-seccodebench

Qwen3‑Coder‑Next slashes through the competition, delivering 10× the throughput of Claude‑Opus‑4.5 on SecCodeBench’s repository‑level tasks. The open‑source model not only speeds up AI code generation but also boosts vulnerability detection. Dive into the benchmark details and see why it’s a game‑changer for secure coding. #Qwen3CoderNext #SecCodeBench #LLMBenchmarking #OpenSourceAI

🔗 https://aidailypost.com/news/qwen3-coder-next-10-throughput-beats-claudeopus45-seccodebench

Anthropic just rolled out Claude Code at $200/month, while the new Claude 4 version climbs to the top of Berkeley’s tool‑calling leaderboard, beating open‑source rivals. Find out how Claude 4’s function‑calling shines and why Goose stays free. #Claude4 #FunctionCalling #BerkeleyLeaderboard #LLMBenchmarking

🔗 https://aidailypost.com/news/claude-code-usd-200mo-goose-free-claude-4-tops-berkeley-toolcalling

Discover how CRITICBENCH tests AI by sampling “convincing wrong answers” to reveal subtle flaws in model reasoning and accuracy. https://hackernoon.com/why-almost-right-answers-are-the-hardest-test-for-ai #llmbenchmarking

Discover how CRITICBENCH tests AI by sampling “convincing wrong answers” to reveal subtle flaws in model reasoning and accuracy. https://hackernoon.com/why-almost-right-answers-are-the-hardest-test-for-ai #llmbenchmarking

Discover how CRITICBENCH tests AI by sampling “convincing wrong answers” to reveal subtle flaws in model reasoning and accuracy. https://hackernoon.com/why-almost-right-answers-are-the-hardest-test-for-ai #llmbenchmarking

Discover how CRITICBENCH tests AI by sampling “convincing wrong answers” to reveal subtle flaws in model reasoning and accuracy. https://hackernoon.com/why-almost-right-answers-are-the-hardest-test-for-ai #llmbenchmarking

Discover how CRITICBENCH tests AI by sampling “convincing wrong answers” to reveal subtle flaws in model reasoning and accuracy. https://hackernoon.com/why-almost-right-answers-are-the-hardest-test-for-ai #llmbenchmarking

Inside CriticBench: How Google’s PaLM-2 models generate benchmark data for GSM8K, HumanEval, and TruthfulQA with open, transparent methods. https://hackernoon.com/why-criticbench-refuses-gpt-and-llama-for-data-generation #llmbenchmarking

Inside CriticBench: How Google’s PaLM-2 models generate benchmark data for GSM8K, HumanEval, and TruthfulQA with open, transparent methods. https://hackernoon.com/why-criticbench-refuses-gpt-and-llama-for-data-generation #llmbenchmarking

Inside CriticBench: How Google’s PaLM-2 models generate benchmark data for GSM8K, HumanEval, and TruthfulQA with open, transparent methods. https://hackernoon.com/why-criticbench-refuses-gpt-and-llama-for-data-generation #llmbenchmarking

Inside CriticBench: How Google’s PaLM-2 models generate benchmark data for GSM8K, HumanEval, and TruthfulQA with open, transparent methods. https://hackernoon.com/why-criticbench-refuses-gpt-and-llama-for-data-generation #llmbenchmarking

Inside CriticBench: How Google’s PaLM-2 models generate benchmark data for GSM8K, HumanEval, and TruthfulQA with open, transparent methods. https://hackernoon.com/why-criticbench-refuses-gpt-and-llama-for-data-generation #llmbenchmarking

Discover CRITICBENCH, the open benchmark comparing GPT-4, PaLM-2, and LLaMA on reasoning, coding, and truth-based critique tasks.
https://hackernoon.com/why-smaller-llms-fail-at-critical-thinking #llmbenchmarking

Discover CRITICBENCH, the open benchmark comparing GPT-4, PaLM-2, and LLaMA on reasoning, coding, and truth-based critique tasks.
https://hackernoon.com/why-smaller-llms-fail-at-critical-thinking #llmbenchmarking

Discover CRITICBENCH, the open benchmark comparing GPT-4, PaLM-2, and LLaMA on reasoning, coding, and truth-based critique tasks.
https://hackernoon.com/why-smaller-llms-fail-at-critical-thinking #llmbenchmarking

Discover CRITICBENCH, the open benchmark comparing GPT-4, PaLM-2, and LLaMA on reasoning, coding, and truth-based critique tasks.
https://hackernoon.com/why-smaller-llms-fail-at-critical-thinking #llmbenchmarking

Discover CRITICBENCH, the open benchmark comparing GPT-4, PaLM-2, and LLaMA on reasoning, coding, and truth-based critique tasks.
https://hackernoon.com/why-smaller-llms-fail-at-critical-thinking #llmbenchmarking

Can AI critique itself? This study shows how self-check improves ChatGPT, GPT-4, and PaLM-2 accuracy on benchmark tasks. https://hackernoon.com/improving-llm-performance-with-self-consistency-and-self-check #llmbenchmarking

Can AI critique itself? This study shows how self-check improves ChatGPT, GPT-4, and PaLM-2 accuracy on benchmark tasks. https://hackernoon.com/improving-llm-performance-with-self-consistency-and-self-check #llmbenchmarking

Can AI critique itself? This study shows how self-check improves ChatGPT, GPT-4, and PaLM-2 accuracy on benchmark tasks. https://hackernoon.com/improving-llm-performance-with-self-consistency-and-self-check #llmbenchmarking

Can AI critique itself? This study shows how self-check improves ChatGPT, GPT-4, and PaLM-2 accuracy on benchmark tasks. https://hackernoon.com/improving-llm-performance-with-self-consistency-and-self-check #llmbenchmarking

Can AI critique itself? This study shows how self-check improves ChatGPT, GPT-4, and PaLM-2 accuracy on benchmark tasks. https://hackernoon.com/improving-llm-performance-with-self-consistency-and-self-check #llmbenchmarking

How well can AI critique its own answers? Explore PaLM-2 results on self-critique, certainty metrics, and why some tasks remain out of reach. https://hackernoon.com/critique-ability-of-large-language-models-self-critique-ability #llmbenchmarking

How well can AI critique its own answers? Explore PaLM-2 results on self-critique, certainty metrics, and why some tasks remain out of reach. https://hackernoon.com/critique-ability-of-large-language-models-self-critique-ability #llmbenchmarking

How well can AI critique its own answers? Explore PaLM-2 results on self-critique, certainty metrics, and why some tasks remain out of reach. https://hackernoon.com/critique-ability-of-large-language-models-self-critique-ability #llmbenchmarking

How well can AI critique its own answers? Explore PaLM-2 results on self-critique, certainty metrics, and why some tasks remain out of reach. https://hackernoon.com/critique-ability-of-large-language-models-self-critique-ability #llmbenchmarking

How well can AI critique its own answers? Explore PaLM-2 results on self-critique, certainty metrics, and why some tasks remain out of reach. https://hackernoon.com/critique-ability-of-large-language-models-self-critique-ability #llmbenchmarking

CRITICBENCH reveals how critique ability scales in LLMs, from self-critique to code evaluation, highlighting when AI becomes a true critic. https://hackernoon.com/why-even-the-best-ai-struggles-at-critiquing-code #llmbenchmarking

CRITICBENCH reveals how critique ability scales in LLMs, from self-critique to code evaluation, highlighting when AI becomes a true critic. https://hackernoon.com/why-even-the-best-ai-struggles-at-critiquing-code #llmbenchmarking

CRITICBENCH reveals how critique ability scales in LLMs, from self-critique to code evaluation, highlighting when AI becomes a true critic. https://hackernoon.com/why-even-the-best-ai-struggles-at-critiquing-code #llmbenchmarking

CRITICBENCH reveals how critique ability scales in LLMs, from self-critique to code evaluation, highlighting when AI becomes a true critic. https://hackernoon.com/why-even-the-best-ai-struggles-at-critiquing-code #llmbenchmarking

CRITICBENCH reveals how critique ability scales in LLMs, from self-critique to code evaluation, highlighting when AI becomes a true critic. https://hackernoon.com/why-even-the-best-ai-struggles-at-critiquing-code #llmbenchmarking

CRITICBENCH refines AI benchmarking with high-quality, certainty-based data selection to build fairer, more differentiable LLM evaluations. https://hackernoon.com/are-your-ai-benchmarks-fooling-you #llmbenchmarking

CRITICBENCH refines AI benchmarking with high-quality, certainty-based data selection to build fairer, more differentiable LLM evaluations. https://hackernoon.com/are-your-ai-benchmarks-fooling-you #llmbenchmarking

CRITICBENCH refines AI benchmarking with high-quality, certainty-based data selection to build fairer, more differentiable LLM evaluations. https://hackernoon.com/are-your-ai-benchmarks-fooling-you #llmbenchmarking

CRITICBENCH refines AI benchmarking with high-quality, certainty-based data selection to build fairer, more differentiable LLM evaluations. https://hackernoon.com/are-your-ai-benchmarks-fooling-you #llmbenchmarking

CRITICBENCH refines AI benchmarking with high-quality, certainty-based data selection to build fairer, more differentiable LLM evaluations. https://hackernoon.com/are-your-ai-benchmarks-fooling-you #llmbenchmarking

CRITICBENCH sets a new standard for evaluating LLM critiques—scalable, generalizable, and focused on quality across diverse tasks.

https://hackernoon.com/constructing-criticbench-scalable-generalizable-and-high-quality-llm-evaluation #llmbenchmarking

CRITICBENCH sets a new standard for evaluating LLM critiques—scalable, generalizable, and focused on quality across diverse tasks.

https://hackernoon.com/constructing-criticbench-scalable-generalizable-and-high-quality-llm-evaluation #llmbenchmarking

CRITICBENCH sets a new standard for evaluating LLM critiques—scalable, generalizable, and focused on quality across diverse tasks.

https://hackernoon.com/constructing-criticbench-scalable-generalizable-and-high-quality-llm-evaluation #llmbenchmarking

CRITICBENCH sets a new standard for evaluating LLM critiques—scalable, generalizable, and focused on quality across diverse tasks.

https://hackernoon.com/constructing-criticbench-scalable-generalizable-and-high-quality-llm-evaluation #llmbenchmarking

CRITICBENCH sets a new standard for evaluating LLM critiques—scalable, generalizable, and focused on quality across diverse tasks.

https://hackernoon.com/constructing-criticbench-scalable-generalizable-and-high-quality-llm-evaluation #llmbenchmarking

CRITICBENCH reveals why large language models struggle with critique and self-criticism, highlighting new methods for AI self-improvement. https://hackernoon.com/criticbench-a-benchmark-for-evaluating-the-critique-abilities-of-llms #llmbenchmarking

CRITICBENCH reveals why large language models struggle with critique and self-criticism, highlighting new methods for AI self-improvement. https://hackernoon.com/criticbench-a-benchmark-for-evaluating-the-critique-abilities-of-llms #llmbenchmarking

CRITICBENCH reveals why large language models struggle with critique and self-criticism, highlighting new methods for AI self-improvement. https://hackernoon.com/criticbench-a-benchmark-for-evaluating-the-critique-abilities-of-llms #llmbenchmarking

CRITICBENCH reveals why large language models struggle with critique and self-criticism, highlighting new methods for AI self-improvement. https://hackernoon.com/criticbench-a-benchmark-for-evaluating-the-critique-abilities-of-llms #llmbenchmarking

CRITICBENCH reveals why large language models struggle with critique and self-criticism, highlighting new methods for AI self-improvement. https://hackernoon.com/criticbench-a-benchmark-for-evaluating-the-critique-abilities-of-llms #llmbenchmarking