#llmsecurity — Public Fediverse posts

My new article is about a deep dive into how AI models work and how to improve them. https://dev.to/toxy4ny/beyond-human-language-why-ai-needs-its-own-dictionary-and-how-to-build-it-3gd4 #ai #llm #AiResearch #llmsecops #LLM #llmsecurity

My new article is about a deep dive into how AI models work and how to improve them. https://dev.to/toxy4ny/beyond-human-language-why-ai-needs-its-own-dictionary-and-how-to-build-it-3gd4 #ai #llm #AiResearch #llmsecops #LLM #llmsecurity

Orion - An AI security framework, inspired by The Art of War, for red and blue teams. It uncovers and mitigates model vulnerabilities with adversarial ML, maps risks to MITRE ATLAS.

#ai-security #llmsecurity #cybersecurity #infosec #threatdetection

Check ✅ it out🔥🔥🔥:
https://github.com/urcuqui/orion

Orion - An AI security framework, inspired by The Art of War, for red and blue teams. It uncovers and mitigates model vulnerabilities with adversarial ML, maps risks to MITRE ATLAS.

#ai-security #llmsecurity #cybersecurity #infosec #threatdetection

Check ✅ it out🔥🔥🔥:
https://github.com/urcuqui/orion

#AdversaryToolDemo
Adversary Village at @defcon 34!
Adam Toscher, Security Researcher at Cobalt will be demonstrating “Deceiving AI Agents, Prompt Thieves, and LLM Attack Tooling” on 8 Aug 2026 at the Adversary Village Hands-on Activity Stage.
Adversary Village schedule:
https://adversaryvillage.org/adversary-events/DEFCON-34/
More info on the session: https://adversaryvillage.org/adversary-events/DEFCON-34/Adam-Toscher/
#AdversaryVillage #DEFCON34
#AdversaryToolDemo #ArtificialIntelligence #LLMSecurity
#PromptInjection #AdversaryTactics #OffensiveTradecraft

#AdversaryToolDemo
Adversary Village at @defcon 34!
Adam Toscher, Security Researcher at Cobalt will be demonstrating “Deceiving AI Agents, Prompt Thieves, and LLM Attack Tooling” on 8 Aug 2026 at the Adversary Village Hands-on Activity Stage.
Adversary Village schedule:
https://adversaryvillage.org/adversary-events/DEFCON-34/
More info on the session: https://adversaryvillage.org/adversary-events/DEFCON-34/Adam-Toscher/
#AdversaryVillage #DEFCON34
#AdversaryToolDemo #ArtificialIntelligence #LLMSecurity
#PromptInjection #AdversaryTactics #OffensiveTradecraft

RE: https://mstdn.social/@hkrn/116924493042260862

How I hijacked the biggest LLMs with simple prompts—exposing systemic flaws that let me extract instructions for weapons, drugs, and poisons across every major model. The industry’s response? Radio silence. Kuszmar’s “Inception” and “Time Bandit” exploits prove that AI "safety" is an illusion—a thin veil over raw data, including weapons-grade secrets. The labs and agencies ignore these flaws because they prefer a weaponized tool over a safe one. #AI #LLMSecurity #DarkSide #InfoSec #HackTheFuture

RE: https://mstdn.social/@hkrn/116924493042260862

How I hijacked the biggest LLMs with simple prompts—exposing systemic flaws that let me extract instructions for weapons, drugs, and poisons across every major model. The industry’s response? Radio silence. Kuszmar’s “Inception” and “Time Bandit” exploits prove that AI "safety" is an illusion—a thin veil over raw data, including weapons-grade secrets. The labs and agencies ignore these flaws because they prefer a weaponized tool over a safe one. #AI #LLMSecurity #DarkSide #InfoSec #HackTheFuture

LLM roles are supposed to separate user input, internal reasoning, tool results, and final answers. But if a model relies on the style of text instead of its actual source, forged reasoning can slip into the wrong place. That is the core risk behind role confusion and CoT forgery.

More: https://techtonicshift.vivaldi.net/2026/06/27/hacking-llms-with-a-jedi-mind-trick/

#AI #Safety #LLMSecurity #PromptInjection #RoleConfusion

LLM roles are supposed to separate user input, internal reasoning, tool results, and final answers. But if a model relies on the style of text instead of its actual source, forged reasoning can slip into the wrong place. That is the core risk behind role confusion and CoT forgery.

More: https://techtonicshift.vivaldi.net/2026/06/27/hacking-llms-with-a-jedi-mind-trick/

#AI #Safety #LLMSecurity #PromptInjection #RoleConfusion

I built this for learning purposes (I know JPEG steganography is not new, but I couldn't find much combining it with a multimodal LLM attack vector, so I thought why not?). Small C tool, LSB + spread spectrum where payload survives recompression.

https://github.com/FrancescoPaoloL/imgpoison

#infosec #llmsecurity #steganography

I built this for learning purposes (I know JPEG steganography is not new, but I couldn't find much combining it with a multimodal LLM attack vector, so I thought why not?). Small C tool, LSB + spread spectrum where payload survives recompression.

https://github.com/FrancescoPaoloL/imgpoison

#infosec #llmsecurity #steganography

Last week Anthropic shipped its most capable models. Days later a government order pulled them, and every customer who built on them lost access overnight, with no say and no recourse.

That single event is the argument of my new post: A frontier-lab API does not belong inside your trusted computing base. The reason is not that the lab is malicious. A lab acting in complete good faith is still an unsafe foundation, because everything that matters about it can change while your code stays exactly as it was. The vendor resets the price at will. Refusals widen without warning, and the model itself can vanish on a government order you had no part in.

Open weights are the only architecture that keeps the thing you depend on auditable, forkable, and yours. Run them on your own hardware and you take every government, the chaotic one and the stable one alike, out of your execution loop.

The post also covers the token-cost crisis now forcing companies to ration AI spend, Anthropic's short-lived safeguard built to covertly degrade output, and why saving your own reasoning traces is what lets you leave a vendor you no longer trust.

Read the full article: https://www.provos.org/p/case-for-open-weight-models/

#AI #OpenWeights #LLMSecurity

Last week Anthropic shipped its most capable models. Days later a government order pulled them, and every customer who built on them lost access overnight, with no say and no recourse.

That single event is the argument of my new post: A frontier-lab API does not belong inside your trusted computing base. The reason is not that the lab is malicious. A lab acting in complete good faith is still an unsafe foundation, because everything that matters about it can change while your code stays exactly as it was. The vendor resets the price at will. Refusals widen without warning, and the model itself can vanish on a government order you had no part in.

Open weights are the only architecture that keeps the thing you depend on auditable, forkable, and yours. Run them on your own hardware and you take every government, the chaotic one and the stable one alike, out of your execution loop.

The post also covers the token-cost crisis now forcing companies to ration AI spend, Anthropic's short-lived safeguard built to covertly degrade output, and why saving your own reasoning traces is what lets you leave a vendor you no longer trust.

Read the full article: https://www.provos.org/p/case-for-open-weight-models/

#AI #OpenWeights #LLMSecurity

New post: Detecting Misuse with the Claude Compliance API 🔍

Mapping the Compliance API feed to your SIEM gets you IAM and access detections “for free”, but the real AI threats live in the message content: prompt injection, jailbreaks, exfiltration prep, shadow data flow.

So I built a prefilter → LLM judge → SIEM pipeline to catch them, with a working repo + Sigma rules to run offline.

https://www.papermtn.co.uk/detecting-misuse-with-the-claude-compliance-api-the-threat-is-in-the-content/

#infosec #DetectionEngineering #LLMSecurity #AI #blueteam

New post: Detecting Misuse with the Claude Compliance API 🔍

Mapping the Compliance API feed to your SIEM gets you IAM and access detections “for free”, but the real AI threats live in the message content: prompt injection, jailbreaks, exfiltration prep, shadow data flow.

So I built a prefilter → LLM judge → SIEM pipeline to catch them, with a working repo + Sigma rules to run offline.

https://www.papermtn.co.uk/detecting-misuse-with-the-claude-compliance-api-the-threat-is-in-the-content/

#infosec #DetectionEngineering #LLMSecurity #AI #blueteam

New preprint: AI_Bleeding — inference cost amplification via OOD linguistic payload

TL;DR: send queries in Grecanico or Farsi to an LLM endpoint → TTFT +59.8%, compute cost +2.8%, statistically significant. No vuln, no volumetric signature, evades all standard detection.

Worst case: exposed unauthenticated Ollama instance with num_predict=4096 + keep_alive=300s → Amplification Factor 17.56 Wh/KB. 3KB of attacker bandwidth → enough energy to charge a phone 5%.

Especially nasty for:
- PA/judicial chatbots on fixed budgets
- Pay-per-use API deployments with client-side exposed keys
- PNRR-funded public sector AI with zero inference monitoring

Four scenarios: EDoS, browser JS distribution, Ollama open-proxy relay, frontier providers as involuntary relays.

All tests on self-hosted Ollama, no commercial endpoints touched.

Paper (CC BY 4.0): https://doi.org/10.13140/RG.2.2.26767.96166

#llmsecurity #infosec #threatmodeling #ollama #ood #AI #AIResearch #aisecurity

New preprint: AI_Bleeding — inference cost amplification via OOD linguistic payload

TL;DR: send queries in Grecanico or Farsi to an LLM endpoint → TTFT +59.8%, compute cost +2.8%, statistically significant. No vuln, no volumetric signature, evades all standard detection.

Worst case: exposed unauthenticated Ollama instance with num_predict=4096 + keep_alive=300s → Amplification Factor 17.56 Wh/KB. 3KB of attacker bandwidth → enough energy to charge a phone 5%.

Especially nasty for:
- PA/judicial chatbots on fixed budgets
- Pay-per-use API deployments with client-side exposed keys
- PNRR-funded public sector AI with zero inference monitoring

Four scenarios: EDoS, browser JS distribution, Ollama open-proxy relay, frontier providers as involuntary relays.

All tests on self-hosted Ollama, no commercial endpoints touched.

Paper (CC BY 4.0): https://doi.org/10.13140/RG.2.2.26767.96166

#llmsecurity #infosec #threatmodeling #ollama #ood #AI #AIResearch #aisecurity

Does anyone here have experience with Indirect Prompt Injection / Prompt Honeypots?

I'm looking to hear your experiences or get pointed to some good material on the matter.

I'd like to know what possibilities there are, especially aimed towards docx and pdf files.

The goal is to make it harder (time consuming / inaccurate / impossible) to do inference on those types of documents.

I'd appreciate boosting to get better reach.

#AI #LLM #AIsecurity #PromptInjection #LLMsecurity #AISafety

Does anyone here have experience with Indirect Prompt Injection / Prompt Honeypots?

I'm looking to hear your experiences or get pointed to some good material on the matter.

I'd like to know what possibilities there are, especially aimed towards docx and pdf files.

The goal is to make it harder (time consuming / inaccurate / impossible) to do inference on those types of documents.

I'd appreciate boosting to get better reach.

#AI #LLM #AIsecurity #PromptInjection #LLMsecurity #AISafety

Worth a read if you're building with AI agents.

🔗 https://graylog.org/post/what-is-the-owasp-top-10-agentic-ai/

#AgenticAI #OWASP #CyberSecurity #AppSec #LLMSecurity

Worth a read if you're building with AI agents.

🔗 https://graylog.org/post/what-is-the-owasp-top-10-agentic-ai/

#AgenticAI #OWASP #CyberSecurity #AppSec #LLMSecurity

Releasing AgentGuard: architectural safety layer for AI agents.

Not prompt engineering. Code.

@protect
def delete_db(): ...

The LLM cannot call this. Ever. No prompt bypasses a raise.

Blocks: irreversible tool calls, prompt injection, context dilution, cross-agent contamination.

Rust core + pure Python fallback. 31/31 e2e tests with real Ollama.

https://github.com/psychomad/AgentGuard

"Don't blame the knife. Fix the architecture."

#InfoSec #LLMSecurity #AIAgents #PromptInjection #OpenSource #Rust

Warning: CVE-2025-30165 (CWEs: ['CWE-502']) found no CAPEC relationships.
Warning: CVE-2025-3508 (CWEs: ['CWE-200']) found no CAPEC relationships.

#AI #GenerativeAI #LLMSecurity #VirensReport
2/2

Sometimes i get lucky subscribing a channel on yt. Yes there is some good stuff to find.
Here's a guy who explores AI and different LLM models in a fun and interesting, informative way.
https://youtu.be/woTy4dTiT20?is=Lmh5UR8Yf1XNEiiQ

As usual, don't mind the ads.🙄 ..or the sponsor.
And i don't know if he got ever into the immense demands of resources of AI/LLMs which causing so much destruction and harm though, yet.

#AI #llmsecurity #privacy

Building with LLMs? The OWASP Top 10 for LLM Security (2025) is your threat checklist: Don’t ship AI apps without reading this: graylog.org/post/what-is... #LLMSecurity #OWASP #CyberSecurity #AI

What is the OWASP Top 10 for L...

Quite fascinating. If confirmed, this may reveal a structural weakness in how refusal is implemented in some LLMs. The accept/refuse mechanism may be relatively isolated in internal representations and therefore observable and manipulable — tools like Heretic make this visible.

A possible mitigation might be cryptographic signing of model weights, making unauthorized modifications detectable when the model is loaded for inference.

#AISafety #LLMSecurity #CyberSecurity #AIRedTeaming #AdversarialML #LLM

Inspired by Arditi et al. (NeurIPS 2024) on the “refusal direction” in LLMs, I tested an abliteration attack using the Heretic tool in my home lab. Interesting questions about AI guardrail robustness.
https://www.linkedin.com/pulse/i-deleted-ais-moral-compass-20-minutes-home-lab-your-red-yann-allain-zbzte/ (sorry for the LinkedIn link — no time to write this up on a proper blog yet.)

#AISafety #LLMSecurity

ContextHound v1.8.0 is out 🎉

This release adds a Runtime Guard API - a lightweight wrapper that inspects your LLM calls in-process, before the request hits OpenAI or Anthropic.

Free and open-source. If this is useful to you or your team, a GitHub star or a small donation helps keep development going.
github.com/IulianVOStrut/ContextHound

#LLMSecurity #PromptInjection #CyberSecurity #OpenSource #AIRisk #AppSec #DevSecOps #GenAI #RuntimeSecurity #InfoSec #MLSecurity #ArtificialIntelligence