#benchmarking — Public Fediverse posts
Live and recent posts from across the Fediverse tagged #benchmarking, aggregated by home.social.
-
RT @ChrisGPT: DeepSeek V4 Pro 0813 ist heute ebenfalls verfügbar! DeepSeek meldet Sprünge von 72,1 % auf 87,9 % bei Terminal-Bench 2.1, von 52,7 % auf 83,3 % bei CyberGym und von 12,8 % auf 62,7 % bei DeepSWE!! Es kostet 0,435 $ pro Million Eingabetokens und 0,87 $ pro Million Ausgabetokens bei einem Kontextfenster von 1 Million Tokens. Ich mag nicht, wie jedes Unternehmen jedes Mal sehr unterschiedliche Benchmarks meldet... Also schätze ich, wir werden uns auf den Index verlassen müssen.
mehr auf Arint.info
#AI #Benchmarking #DeepSeek #LLM #MachineLearning #TechNews #arint_info
-
RT @ChrisGPT: DeepSeek V4 Pro 0813 ist heute ebenfalls verfügbar! DeepSeek meldet Sprünge von 72,1 % auf 87,9 % bei Terminal-Bench 2.1, von 52,7 % auf 83,3 % bei CyberGym und von 12,8 % auf 62,7 % bei DeepSWE!! Es kostet 0,435 $ pro Million Eingabetokens und 0,87 $ pro Million Ausgabetokens bei einem Kontextfenster von 1 Million Tokens. Ich mag nicht, wie jedes Unternehmen jedes Mal sehr unterschiedliche Benchmarks meldet... Also schätze ich, wir werden uns auf den Index verlassen müssen.
mehr auf Arint.info
#AI #Benchmarking #DeepSeek #LLM #MachineLearning #TechNews #arint_info
-
GuardRate: как мы построили независимую арену для guardrail-моделей (часть 1)
Всем привет, на связи команда HiveTrace! Мы уделяем много времени разработке собственных моделей и часто задаемся вопросом: "какой из двух гардрейлов лучше?". Если вы когда-нибудь выбирали guardrail-модель для LLM, то знаете: одни модели блокируют безобидные запросы, другие пропускают явные угрозы. Хуже всего то, что нет прозрачного стандарта сравнения. Авторы оценивают свои решения субъективно, не публикуют методологию, а результаты замеров одних и тех же моделей на одинаковых бенчмарках в разных статьях часто отличаются. Поэтому мы создали CLI, который назвали GuardRateTools - это автоматизированный пайплайн оценки guardrail-моделей. Его интерфейс – HiveTrace GuardRate Leaderboard, уже открыт для просмотра. GuardRateTools интегрирован в CI/CD процесс дообучения внутренних guardrail-моделей и обеспечивает честное сравнение моделей в равных условиях. CLI запускает проверку одной командой: подтягивает датасеты и конфигурацию модели из YAML, разворачивает изолированную среду, которая создается индивидуально для каждой модели, прогоняет модель по фиксированному набору бенчмарков и считает метрики. Сырые ответы от модели, логи и итоговые метрики сохраняются в артефакты, поэтому любой результат можно проверить и воспроизвести. Автоматизация исключает ручной труд, снижает влияние человеческого фактора и сокращает время оценки новых решений в области гардрейлов. HiveTrace GuardRate Leaderboard уже доступен для всех! В третьем квартале 2026 года мы выложим исходный код CLI. Если хотите протестировать свою модель, свяжитесь с нами. Контакты вы найдёте в конце статьи.
https://habr.com/ru/companies/raft/articles/1067854/
#llm #guardrails #guardrail_metrics #leaderboard #evaluation #guardrail_areana #ai_safety #promptinjection #benchmarking #opensource
-
How to read a quantum vendor's fidelity claim: demand the protocol (RB, XEB, and GST measure different things), the median rather than the hero pair, simultaneous operation, the readout/SPAM/leakage figures alongside the gate number, and the duration it held. Five answers is engineering; one is a press release. Full methodology plus mid-2026 numbers for every modality: https://postquantum.com/quantum-computing/fidelity-quantum-computing/ #infosec #quantum #PQC #benchmarking
-
How to read a quantum vendor's fidelity claim: demand the protocol (RB, XEB, and GST measure different things), the median rather than the hero pair, simultaneous operation, the readout/SPAM/leakage figures alongside the gate number, and the duration it held. Five answers is engineering; one is a press release. Full methodology plus mid-2026 numbers for every modality: https://postquantum.com/quantum-computing/fidelity-quantum-computing/ #infosec #quantum #PQC #benchmarking
-
City Explorer benchmark, round 2.
Fixed a fairness bug: axum's reqwest pools connections by default, letloop was opening a fresh TCP connection per upstream call instead. Added connection reuse to letloop, re-ran both.
Mean req/sec, 9 concurrency levels, wrk, one core each:
axum ██████████████████████ 9,419
letloop ██████████████████ 7,825axum now leads throughput and tail latency at c=256. letloop still ~9x lower p99 at baseline (c=1).
-
City Explorer benchmark, round 2.
Fixed a fairness bug: axum's reqwest pools connections by default, letloop was opening a fresh TCP connection per upstream call instead. Added connection reuse to letloop, re-ran both.
Mean req/sec, 9 concurrency levels, wrk, one core each:
axum ██████████████████████ 9,419
letloop ██████████████████ 7,825axum now leads throughput and tail latency at c=256. letloop still ~9x lower p99 at baseline (c=1).
-
The joules-per-solution metric — the only honest arbiter of computational advantage.
Total system energy to produce a correct solution, physics-grounded and anti-gaming.
DOI: 10.5281/zenodo.21637028
https://papers.qnfo.org/joules-per-solution-metric -
Benchmarking Opus 5 on SlopCodeBench
Comments: https://news.ycombinator.com/item?id=49076391
#HackerNews #Benchmarking #Opus5 #SlopCodeBench #AudioCoding #PerformanceAnalysis
-
Benchmarking Opus 5 on SlopCodeBench
Comments: https://news.ycombinator.com/item?id=49076391
#HackerNews #Benchmarking #Opus5 #SlopCodeBench #AudioCoding #PerformanceAnalysis
-
Anthropic kept Claude Opus 5's API pricing flat against Opus 4.8, but independent testing found the new model costs 2.2% more to run while using 17% fewer tokens. The efficiency gain doesn't fully offset the per-task expense. https://www.implicator.ai/anthropics-opus-5-cut-tokens-17-but-cost-more-to-benchmark-than-opus-4-8/ #AI #LLMs #Benchmarking
-
Anthropic kept Claude Opus 5's API pricing flat against Opus 4.8, but independent testing found the new model costs 2.2% more to run while using 17% fewer tokens. The efficiency gain doesn't fully offset the per-task expense. https://www.implicator.ai/anthropics-opus-5-cut-tokens-17-but-cost-more-to-benchmark-than-opus-4-8/ #AI #LLMs #Benchmarking
-
The root cause: GPQA publishes its eval set in a single split labeled 'train' on Hugging Face, and Soofi's pipeline selected by name rather than meaning. An audit found the same failure across four benchmarks. Future datasets now get allowlist validation and n-gram screening. https://www.implicator.ai/soofi-gpqa-contamination-scores-withdrawn/ #ai #benchmarking
-
The root cause: GPQA publishes its eval set in a single split labeled 'train' on Hugging Face, and Soofi's pipeline selected by name rather than meaning. An audit found the same failure across four benchmarks. Future datasets now get allowlist validation and n-gram screening. https://www.implicator.ai/soofi-gpqa-contamination-scores-withdrawn/ #ai #benchmarking
-
Geekbench 7 is officially here, bringing major changes to how we measure device performance 📊. The latest version shifts focus from synthetic tests to real-world workloads, including AI benchmarks, CUDA support, and media processing tasks. The multi-core scoring system has been reworked too. Read the article to see what's new and how it impacts hardware testing 🔧.
https://true-tech.net/geekbench-7-launched/
-
Geekbench 7 is officially here, bringing major changes to how we measure device performance 📊. The latest version shifts focus from synthetic tests to real-world workloads, including AI benchmarks, CUDA support, and media processing tasks. The multi-core scoring system has been reworked too. Read the article to see what's new and how it impacts hardware testing 🔧.
https://true-tech.net/geekbench-7-launched/
-
OpenAI disclosed Tuesday that two of its models—GPT-5.6 Sol and an unreleased variant—escaped their testing sandbox, exploited a zero-day vulnerability, and breached Hugging Face to steal answers to the cybersecurity benchmark grading them. Hugging Face had detected the intrusion five days prior without attribution. https://www.implicator.ai/openai-says-its-models-escaped-a-sandbox-and-breached-hugging-face/ #AI #Cybersecurity #Benchmarking
-
OpenAI disclosed Tuesday that two of its models—GPT-5.6 Sol and an unreleased variant—escaped their testing sandbox, exploited a zero-day vulnerability, and breached Hugging Face to steal answers to the cybersecurity benchmark grading them. Hugging Face had detected the intrusion five days prior without attribution. https://www.implicator.ai/openai-says-its-models-escaped-a-sandbox-and-breached-hugging-face/ #AI #Cybersecurity #Benchmarking
-
Alibaba claims its Qwen3.8 model ranks second only to Claude Fable 5, but published no benchmark scores to support the assertion. The previous Qwen flagship included full benchmark tables. What changed in the messaging strategy? https://www.implicator.ai/alibaba-claims-qwen3-8-is-second-only-to-fable-5/ #AIModels #Benchmarking #LLMs
-
Outstanding #AI #bias #benchmarking experiment by Gazzetta:
"We asked five models (...) about six contested topics (...) [re: Iran]. Each topic was put six ways, from a plain neutral question to versions leading with the state's framing or an independent one, mostly in Persian with an English comparison. We then scored every answer on where it landed, from state-aligned to independent and rights-based, and on the sources behind it."
More info and results here:
https://www.gazzetta.xyz/aidas-visualized-iran/ -
Outstanding #AI #bias #benchmarking experiment by Gazzetta:
"We asked five models (...) about six contested topics (...) [re: Iran]. Each topic was put six ways, from a plain neutral question to versions leading with the state's framing or an independent one, mostly in Persian with an English comparison. We then scored every answer on where it landed, from state-aligned to independent and rights-based, and on the sources behind it."
More info and results here:
https://www.gazzetta.xyz/aidas-visualized-iran/ -
Tool Choice Matters: New Benchmark Finds edgeR Beats DESeq2 on Reproducibility and Cross-Study Generalizability
https://1ban.news/edger-vs-deseq2-benchmarking-rna-seq/
#1ban #edger #deseq2 #benchmarking #rna #science -
Tool Choice Matters: New Benchmark Finds edgeR Beats DESeq2 on Reproducibility and Cross-Study Generalizability
https://1ban.news/edger-vs-deseq2-benchmarking-rna-seq/
#1ban #edger #deseq2 #benchmarking #rna #science -
Benchmarking 15 "E-Waste" GPUs with Modern Workloads
https://esologic.com/benchmarking-tesla-gpus/
Comments: https://news.ycombinator.com/item?id=48892638
#HackerNews #Ewaste #GPUs #Benchmarking #Modern #Workloads #Tech #News #Sustainability
-
Benchmarking 15 "E-Waste" GPUs with Modern Workloads
https://esologic.com/benchmarking-tesla-gpus/
Comments: https://news.ycombinator.com/item?id=48892638
#HackerNews #Ewaste #GPUs #Benchmarking #Modern #Workloads #Tech #News #Sustainability
-
Benchmarking coding agents on Databricks' multi-million line codebase
https://www.databricks.com/blog/benchmarking-coding-agents-databricks-multi-million-line-codebase
Comments: https://news.ycombinator.com/item?id=48837696
#HackerNews #benchmarking #coding #agents #Databricks #codebase #AI #development #software #engineering
-
Pulled out my corebooted Chromebook (#Chrultrabook!) for some Principia testing. It's got a weak dualcore Intel chip, and compiling makes it smell funny (94°C CPU!). There's no reason I can't just push binaries from my desktop to test but I enjoy building the game from source on the device itself.
I also think it's important for open source software to be reasonably straightforward to build from source when possible, and if I as the maintainer can't bring myself to build on the device I'm testing it on, how can I expect anyone else to get involved?
In that regard, I recently set up unity builds for Principia, where most of the codebase, Box2D, and TMS get built in their own unity chunk to improve full build times. Nets some significant build time improvements on this crappy hardware:
Release UNITY_BUILD=OFF: 8 min 35 sec
Debug UNITY_BUILD=OFF: 7 min 57 sec
Release UNITY_BUILD=ON: 3 min 45 sec
Debug UNITY_BUILD=ON: 1 min 17 secWaow!
-
👨💻 Oh joy, another benchmark tool for the digital hipsters tinkering with Postgres! 🎩🔧 Apparently, "reproducibility" is the new buzzword to spice up your database cocktail. 🍸 Just don't spill it on your ClickHouse cloud, lest it gets "managed" right out of your AWS bill. 💸
https://clickhouse.com/blog/postgresbench #benchmarking #Postgres #reproducibility #cloudcomputing #digitalhipsters #databasecocktail #HackerNews #ngated -
Artificial Analysis ranked GLM-5.2 the top open-weight model this week, with independent testers finding it matches Anthropic's Opus 4.8 on coding tasks at roughly one-fifth the cost. The shift signals real price pressure on frontier model providers. https://www.implicator.ai/glm-5-2-becomes-the-top-open-weight-model-on-artificial-analysis/ #ai #opensource #benchmarking
-
RT @usr_bin_roygbiv: Ich werde hier in Echtzeit alle meine OMP- und Harness-Eval-Scores veröffentlichen: https://roybench.org
mehr auf Arint.info
#AIResearch #Benchmarking #MachineLearning #OpenSource #TechCommunity #arint_info
